[2022-12-20 10:11:56,294] [WARNING] [runner.py:179:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2022-12-20 10:11:56,305] [INFO] [runner.py:508:main] cmd = /home/milan/hf_env/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29560 run_speech_recognition_seq2seq_streaming.py --deepspeed=ds_config.json --model_name_or_path=openai/whisper-large-v2 --dataset_name=mozilla-foundation/common_voice_11_0 --dataset_config_name=cs --language=czech --train_split_name=train+validation --eval_split_name=test --model_index_name=Whisper Large-v2 Czech CV11 v2 --max_steps=5000 --output_dir=./ --per_device_train_batch_size=32 --per_device_eval_batch_size=8 --gradient_accumulation_steps=2 --logging_steps=25 --learning_rate=1e-5 --warmup_steps=500 --evaluation_strategy=steps --eval_steps=1000 --save_strategy=steps --save_steps=1000 --generation_max_length=225 --length_column_name=input_length --max_duration_in_seconds=30 --text_column_name=sentence --freeze_feature_encoder=False --report_to=tensorboard --metric_for_best_model=wer --greater_is_better=False --load_best_model_at_end --gradient_checkpointing --fp16 --overwrite_output_dir --do_train --do_eval --predict_with_generate --do_normalize_eval --streaming=False --use_auth_token --push_to_hub
[2022-12-20 10:11:57,861] [INFO] [launch.py:142:main] WORLD INFO DICT: {'localhost': [0]}
[2022-12-20 10:11:57,861] [INFO] [launch.py:148:main] nnodes=1, num_local_procs=1, node_rank=0
[2022-12-20 10:11:57,861] [INFO] [launch.py:161:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2022-12-20 10:11:57,861] [INFO] [launch.py:162:main] dist_world_size=1
[2022-12-20 10:11:57,861] [INFO] [launch.py:164:main] Setting CUDA_VISIBLE_DEVICES=0
[2022-12-20 10:12:02,018] [INFO] [comm.py:654:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
12/20/2022 10:12:02 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: True
12/20/2022 10:12:02 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=ds_config.json,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_steps=1000,
evaluation_strategy=steps,
fp16=True,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
generation_max_length=225,
generation_num_beams=None,
gradient_accumulation_steps=2,
gradient_checkpointing=True,
greater_is_better=False,
group_by_length=False,
half_precision_backend=auto,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=1e-05,
length_column_name=input_length,
load_best_model_at_end=True,
local_rank=0,
log_level=passive,
log_level_replica=passive,
log_on_each_node=True,
logging_dir=./runs/Dec20_10-12-02_129-146-123-136,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=25,
logging_strategy=steps,
lr_scheduler_type=linear,
max_grad_norm=1.0,
max_steps=5000,
metric_for_best_model=wer,
mp_parameters=,
no_cuda=False,
num_train_epochs=3.0,
optim=adamw_hf,
optim_args=None,
output_dir=./,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=32,
predict_with_generate=True,
prediction_loss_only=False,
push_to_hub=True,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=./,
save_on_each_node=False,
save_steps=1000,
save_strategy=steps,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
sortish_sampler=False,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=500,
weight_decay=0.0,
xpu_backend=None,
)
12/20/2022 10:12:02 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=ds_config.json,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_steps=1000,
evaluation_strategy=steps,
fp16=True,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
generation_max_length=225,
generation_num_beams=None,
gradient_accumulation_steps=2,
gradient_checkpointing=True,
greater_is_better=False,
group_by_length=False,
half_precision_backend=auto,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=1e-05,
length_column_name=input_length,
load_best_model_at_end=True,
local_rank=0,
log_level=passive,
log_level_replica=passive,
log_on_each_node=True,
logging_dir=./runs/Dec20_10-12-02_129-146-123-136,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=25,
logging_strategy=steps,
lr_scheduler_type=linear,
max_grad_norm=1.0,
max_steps=5000,
metric_for_best_model=wer,
mp_parameters=,
no_cuda=False,
num_train_epochs=3.0,
optim=adamw_hf,
optim_args=None,
output_dir=./,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=32,
predict_with_generate=True,
prediction_loss_only=False,
push_to_hub=True,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=./,
save_on_each_node=False,
save_steps=1000,
save_strategy=steps,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
sortish_sampler=False,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=500,
weight_decay=0.0,
xpu_backend=None,
)
12/20/2022 10:12:04 - INFO - datasets.info - Loading Dataset Infos from /home/milan/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_11_0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
12/20/2022 10:12:04 - INFO - datasets.builder - Overwrite dataset info from restored data version.
12/20/2022 10:12:04 - INFO - datasets.info - Loading Dataset info from /home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
12/20/2022 10:12:04 - WARNING - datasets.builder - Found cached dataset common_voice_11_0 (/home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f)
12/20/2022 10:12:04 - INFO - datasets.info - Loading Dataset info from /home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
12/20/2022 10:12:06 - INFO - datasets.info - Loading Dataset Infos from /home/milan/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_11_0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
12/20/2022 10:12:06 - INFO - datasets.builder - Overwrite dataset info from restored data version.
12/20/2022 10:12:06 - INFO - datasets.info - Loading Dataset info from /home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
12/20/2022 10:12:06 - WARNING - datasets.builder - Found cached dataset common_voice_11_0 (/home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f)
12/20/2022 10:12:06 - INFO - datasets.info - Loading Dataset info from /home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
12/20/2022 10:12:08 - INFO - datasets.info - Loading Dataset Infos from /home/milan/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_11_0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
12/20/2022 10:12:08 - INFO - datasets.builder - Overwrite dataset info from restored data version.
12/20/2022 10:12:08 - INFO - datasets.info - Loading Dataset info from /home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
12/20/2022 10:12:08 - WARNING - datasets.builder - Found cached dataset common_voice_11_0 (/home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f)
12/20/2022 10:12:08 - INFO - datasets.info - Loading Dataset info from /home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
[INFO|configuration_utils.py:654] 2022-12-20 10:12:08,655 >> loading configuration file config.json from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/config.json
[INFO|configuration_utils.py:706] 2022-12-20 10:12:08,659 >> Model config WhisperConfig {
  "_name_or_path": "openai/whisper-large-v2",
  "activation_dropout": 0.0,
  "activation_function": "gelu",
  "architectures": [
    "WhisperForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "begin_suppress_tokens": [
    220,
    50257
  ],
  "bos_token_id": 50257,
  "d_model": 1280,
  "decoder_attention_heads": 20,
  "decoder_ffn_dim": 5120,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 32,
  "decoder_start_token_id": 50258,
  "dropout": 0.0,
  "encoder_attention_heads": 20,
  "encoder_ffn_dim": 5120,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 32,
  "eos_token_id": 50257,
  "forced_decoder_ids": [
    [
      1,
      50259
    ],
    [
      2,
      50359
    ],
    [
      3,
      50363
    ]
  ],
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "max_length": 448,
  "max_source_positions": 1500,
  "max_target_positions": 448,
  "model_type": "whisper",
  "num_hidden_layers": 32,
  "num_mel_bins": 80,
  "pad_token_id": 50257,
  "scale_embedding": false,
  "suppress_tokens": [
    1,
    2,
    7,
    8,
    9,
    10,
    14,
    25,
    26,
    27,
    28,
    29,
    31,
    58,
    59,
    60,
    61,
    62,
    63,
    90,
    91,
    92,
    93,
    359,
    503,
    522,
    542,
    873,
    893,
    902,
    918,
    922,
    931,
    1350,
    1853,
    1982,
    2460,
    2627,
    3246,
    3253,
    3268,
    3536,
    3846,
    3961,
    4183,
    4667,
    6585,
    6647,
    7273,
    9061,
    9383,
    10428,
    10929,
    11938,
    12033,
    12331,
    12562,
    13793,
    14157,
    14635,
    15265,
    15618,
    16553,
    16604,
    18362,
    18956,
    20075,
    21675,
    22520,
    26130,
    26161,
    26435,
    28279,
    29464,
    31650,
    32302,
    32470,
    36865,
    42863,
    47425,
    49870,
    50254,
    50258,
    50360,
    50361,
    50362
  ],
  "torch_dtype": "float32",
  "transformers_version": "4.26.0.dev0",
  "use_cache": true,
  "vocab_size": 51865
}

[INFO|feature_extraction_utils.py:464] 2022-12-20 10:12:08,896 >> loading configuration file preprocessor_config.json from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/preprocessor_config.json
[INFO|feature_extraction_utils.py:501] 2022-12-20 10:12:08,912 >> Feature extractor WhisperFeatureExtractor {
  "chunk_length": 30,
  "feature_extractor_type": "WhisperFeatureExtractor",
  "feature_size": 80,
  "hop_length": 160,
  "mel_filters": [
    [
      -0.0,
      0.02486259490251541,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.001990821911022067,
      0.022871771827340126,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.003981643822044134,
      0.02088095061480999,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0059724655002355576,
      0.018890129402279854,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.007963287644088268,
      0.01689930632710457,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.009954108856618404,
      0.014908484183251858,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.011944931000471115,
      0.012917662039399147,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.013935752213001251,
      0.010926840826869011,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.015926575288176537,
      0.0089360186830163,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.017917396500706673,
      0.006945197004824877,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01990821771323681,
      0.004954374860972166,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.021899040788412094,
      0.0029635531827807426,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.02388986200094223,
      0.0009727313299663365,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.025880683213472366,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.025835324078798294,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0010180906392633915,
      0.023844502866268158,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.003008912317454815,
      0.021853681653738022,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.004999734461307526,
      0.019862858578562737,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.006990555673837662,
      0.0178720373660326,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.008981377817690372,
      0.015881216153502464,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.010972199961543083,
      0.013890394009649754,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01296302117407322,
      0.011899571865797043,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01495384331792593,
      0.009908749721944332,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01694466546177864,
      0.007917927578091621,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.018935488536953926,
      0.005927106365561485,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.020874010398983955,
      0.004040425643324852,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.022114217281341553,
      0.0033186059445142746,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.02173672430217266,
      0.0036109676584601402,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.020497702062129974,
      0.004762193653732538,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.018486659973859787,
      0.006592618301510811,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01585603691637516,
      0.00896277092397213,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.012738768011331558,
      0.011751330457627773,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.009250369854271412,
      0.014853144995868206,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.005490840878337622,
      0.018177473917603493,
      0.0028155462350696325,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0015463664894923568,
      0.01632951945066452,
      0.007420188747346401,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.011181050911545753,
      0.012018864043056965,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.006065350491553545,
      0.016561277210712433,
      0.004360878840088844,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0010297985281795263,
      0.012770536355674267,
      0.009707189165055752,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.006986402906477451,
      0.01485429983586073,
      0.004391219466924667,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.001418047584593296,
      0.011486922390758991,
      0.010089744813740253,
      0.00040022286702878773,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.005411104764789343,
      0.014735566452145576,
      0.006518189795315266,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00827841367572546,
      0.012277561239898205,
      0.00396781275048852,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.002187808509916067,
      0.010184479877352715,
      0.00998187530785799,
      0.0022864851634949446,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00386943481862545,
      0.011274894699454308,
      0.008466221392154694,
      0.0013397691072896123,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.004820294212549925,
      0.011678251437842846,
      0.007608682848513126,
      0.0010091039584949613,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.005156961735337973,
      0.011507894843816757,
      0.007301822770386934,
      0.0011901655234396458,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.004982104524970055,
      0.010863498784601688,
      0.007451189681887627,
      0.001791381393559277,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.004385921638458967,
      0.009832492098212242,
      0.007973956875503063,
      0.002732589840888977,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0034474546555429697,
      0.008491347543895245,
      0.008797688409686089,
      0.00394382793456316,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0022357646375894547,
      0.0069067515432834625,
      0.009859241545200348,
      0.005364237818866968,
      0.0008692338014952838,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0008110002381727099,
      0.005136650986969471,
      0.00946230161935091,
      0.0069410777650773525,
      0.0027783995028585196,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.003231203882023692,
      0.007237049750983715,
      0.00862883497029543,
      0.004773912951350212,
      0.0009189908159896731,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.001233637798577547,
      0.0049433219246566296,
      0.008653006516397,
      0.006818502210080624,
      0.003248583758249879,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0026164355222135782,
      0.006051854696124792,
      0.008880467154085636,
      0.005574479699134827,
      0.002268492942675948,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0002863667905330658,
      0.003467798000201583,
      0.0066492292098701,
      0.00787146482616663,
      0.004809896927326918,
      0.0017483289120718837,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0009245910914614797,
      0.0038708120118826628,
      0.00681703258305788,
      0.007283343467861414,
      0.004448124207556248,
      0.0016129047144204378,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0011703289346769452,
      0.003898728871718049,
      0.006627128925174475,
      0.0070473202504217625,
      0.004421714693307877,
      0.0017961094854399562,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0010892992140725255,
      0.003615982597693801,
      0.006142666097730398,
      0.007102936040610075,
      0.004671447444707155,
      0.002239959081634879,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0007392280967906117,
      0.0030791081953793764,
      0.005418988410383463,
      0.007397185545414686,
      0.005145462695509195,
      0.002893739379942417,
      0.0006420162972062826,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00017068670422304422,
      0.0023375742603093386,
      0.004504461772739887,
      0.0066713495180010796,
      0.005798479542136192,
      0.003713231300935149,
      0.0016279831761494279,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0014345343224704266,
      0.0034412189852446318,
      0.005447904113680124,
      0.006591092795133591,
      0.004660011734813452,
      0.002728930441662669,
      0.0007978491485118866,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0004075043834745884,
      0.002265830524265766,
      0.004124156199395657,
      0.005982482805848122,
      0.005700822453945875,
      0.003912510350346565,
      0.0021241982467472553,
      0.0003358862304594368,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0010099108330905437,
      0.002730846870690584,
      0.004451782442629337,
      0.006172718480229378,
      0.005150905344635248,
      0.0034948070533573627,
      0.0018387088784947991,
      0.0001826105872169137,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0012943691108375788,
      0.002888072282075882,
      0.004481775686144829,
      0.006075479090213776,
      0.0048866597935557365,
      0.003353001084178686,
      0.0018193417927250266,
      0.00028568264679051936,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0013131388695910573,
      0.0027890161145478487,
      0.004264893010258675,
      0.0057407706044614315,
      0.004859979264438152,
      0.0034397069830447435,
      0.0020194342359900475,
      0.0005991620710119605,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0011121684219688177,
      0.002478930866345763,
      0.0038456933107227087,
      0.0052124555222690105,
      0.005028639920055866,
      0.0037133716978132725,
      0.002398103242740035,
      0.0010828346712514758,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0007317548734135926,
      0.0019974694587290287,
      0.003263183869421482,
      0.004528898745775223,
      0.005355686880648136,
      0.004137659445405006,
      0.0029196315445005894,
      0.0017016039928421378,
      0.0004835762665607035,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00020713974663522094,
      0.0013792773243039846,
      0.0025514145381748676,
      0.003723552217707038,
      0.004895689897239208,
      0.004680895246565342,
      0.0035529187880456448,
      0.0024249425623565912,
      0.0012969663366675377,
      0.00016899015463422984,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0006545265205204487,
      0.0017400053329765797,
      0.0028254841454327106,
      0.003910962492227554,
      0.004996441304683685,
      0.0042709787376224995,
      0.003226396394893527,
      0.002181813819333911,
      0.0011372314766049385,
      9.264905384043232e-05,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.000854626705404371,
      0.001859853626228869,
      0.002865080488845706,
      0.003870307235047221,
      0.00487553421407938,
      0.00408313749358058,
      0.003115783678367734,
      0.0021484296303242445,
      0.001181075582280755,
      0.0002137213887181133,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0008483415003865957,
      0.0017792496364563704,
      0.0027101580053567886,
      0.0036410661414265633,
      0.004571974277496338,
      0.004079728852957487,
      0.003183893393725157,
      0.002288057701662183,
      0.0013922222424298525,
      0.0004963868414051831,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0006716204807162285,
      0.0015337044605985284,
      0.002395788673311472,
      0.0032578727696090937,
      0.004119956865906715,
      0.004227725323289633,
      0.0033981208689510822,
      0.0025685166474431753,
      0.0017389123095199466,
      0.0009093079133890569,
      7.970355363795534e-05,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0003559796023182571,
      0.0011543278815224767,
      0.0019526762189343572,
      0.002751024439930916,
      0.0035493727773427963,
      0.004347721114754677,
      0.0037299629766494036,
      0.002961693098768592,
      0.00219342322088778,
      0.0014251532265916467,
      0.0006568834069184959,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0006682946113869548,
      0.0014076193328946829,
      0.0021469437051564455,
      0.002886268775910139,
      0.0036255933810025454,
      0.004154576454311609,
      0.0034431067761033773,
      0.0027316368650645018,
      0.0020201667211949825,
      0.0013086966937407851,
      0.0005972267827019095,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      9.926508937496692e-05,
      0.0007839298341423273,
      0.001468594535253942,
      0.0021532592363655567,
      0.0028379240538924932,
      0.0035225888714194298,
      0.0039915177039802074,
      0.0033326479606330395,
      0.002673778682947159,
      0.002014909405261278,
      0.0013560398947447538,
      0.0006971705006435513,
      3.8301113818306476e-05,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00010181095422012731,
      0.0007358568836934865,
      0.0013699028640985489,
      0.0020039486698806286,
      0.002637994708493352,
      0.0032720407471060753,
      0.003906086552888155,
      0.0033682563807815313,
      0.0027580985333770514,
      0.002147940918803215,
      0.0015377833042293787,
      0.0009276255150325596,
      0.000317467754939571,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0005530364578589797,
      0.0011402058880776167,
      0.0017273754347115755,
      0.0023145449813455343,
      0.002901714527979493,
      0.003488884074613452,
      0.003523340215906501,
      0.002958292607218027,
      0.002393245231360197,
      0.0018281979719176888,
      0.001263150479644537,
      0.0006981031037867069,
      0.0001330557424807921,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0002608386566862464,
      0.0008045974536798894,
      0.0013483562506735325,
      0.0018921148730441928,
      0.0024358737282454967,
      0.002979632467031479,
      0.003523391205817461,
      0.003251380519941449,
      0.0027281083166599274,
      0.002204835880547762,
      0.001681563793681562,
      0.001158291706815362,
      0.0006350195035338402,
      0.00011174729297636077,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0003849811910185963,
      0.0008885387214832008,
      0.001392096164636314,
      0.0018956535495817661,
      0.00239921105094254,
      0.002902768552303314,
      0.0034063260536640882,
      0.003132763085886836,
      0.0026481777895241976,
      0.0021635922603309155,
      0.0016790067311376333,
      0.0011944210855290294,
      0.0007098356145434082,
      0.00022525011445395648,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.000366741674952209,
      0.0008330700220540166,
      0.0012993983691558242,
      0.0017657268326729536,
      0.0022320549469441175,
      0.002698383294045925,
      0.0031647118739783764,
      0.003141313325613737,
      0.002692554146051407,
      0.0022437951993197203,
      0.00179503601975739,
      0.0013462770730257034,
      0.000897518009878695,
      0.0004487590049393475,
      0.0
    ]
  ],
  "n_fft": 400,
  "n_samples": 480000,
  "nb_max_frames": 3000,
  "padding_side": "right",
  "padding_value": 0.0,
  "processor_class": "WhisperProcessor",
  "return_attention_mask": false,
  "sampling_rate": 16000
}

[INFO|tokenization_utils_base.py:1799] 2022-12-20 10:12:09,157 >> loading file vocab.json from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/vocab.json
[INFO|tokenization_utils_base.py:1799] 2022-12-20 10:12:09,157 >> loading file tokenizer.json from cache at None
[INFO|tokenization_utils_base.py:1799] 2022-12-20 10:12:09,157 >> loading file merges.txt from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/merges.txt
[INFO|tokenization_utils_base.py:1799] 2022-12-20 10:12:09,157 >> loading file normalizer.json from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/normalizer.json
[INFO|tokenization_utils_base.py:1799] 2022-12-20 10:12:09,157 >> loading file added_tokens.json from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/added_tokens.json
[INFO|tokenization_utils_base.py:1799] 2022-12-20 10:12:09,157 >> loading file special_tokens_map.json from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/special_tokens_map.json
[INFO|tokenization_utils_base.py:1799] 2022-12-20 10:12:09,157 >> loading file tokenizer_config.json from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/tokenizer_config.json
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|endoftext|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|startoftranscript|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|en|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|zh|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|de|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|es|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|ru|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|ko|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|fr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|ja|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|pt|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|tr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|pl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|ca|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|nl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|ar|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|sv|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|it|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|id|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|hi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|fi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|vi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,220 >> Adding <|iw|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|uk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|el|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|ms|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|cs|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|ro|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|da|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|hu|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|ta|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|no|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|th|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|ur|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|hr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|bg|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|lt|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|la|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|mi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|ml|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|cy|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|sk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|te|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|fa|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|lv|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|bn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|sr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|az|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|sl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|kn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|et|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|mk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|br|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|eu|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|is|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|hy|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|ne|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|mn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,221 >> Adding <|bs|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|kk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|sq|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|sw|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|gl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|mr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|pa|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|si|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|km|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|sn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|yo|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|so|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|af|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|oc|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|ka|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|be|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|tg|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|sd|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|gu|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|am|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|yi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|lo|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|uz|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|fo|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|ht|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|ps|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|tk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|nn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|mt|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|sa|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|lb|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|my|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|bo|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,222 >> Adding <|tl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|mg|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|as|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|tt|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|haw|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|ln|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|ha|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|ba|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|jw|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|su|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|translate|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|transcribe|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|startoflm|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|startofprev|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|nocaptions|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:12:09,223 >> Adding <|notimestamps|> to the vocabulary
[INFO|modeling_utils.py:2251] 2022-12-20 10:12:09,227 >> loading weights file pytorch_model.bin from cache at /home/milan/.cache/huggingface/hub/models--openai--whisper-large-v2/snapshots/e823955b7861a1d66fef509b8601ada6d7762c03/pytorch_model.bin
[INFO|modeling_utils.py:2806] 2022-12-20 10:12:25,819 >> All model checkpoint weights were used when initializing WhisperForConditionalGeneration.

[INFO|modeling_utils.py:2814] 2022-12-20 10:12:25,819 >> All the weights of WhisperForConditionalGeneration were initialized from the model checkpoint at openai/whisper-large-v2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use WhisperForConditionalGeneration for predictions without further training.

  0%|          | 0/236 [00:00<?, ?ba/s]12/20/2022 10:12:29 - INFO - datasets.arrow_dataset - Caching processed dataset at /home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f/cache-1c16fd55b44eb2f3.arrow

  0%|          | 1/236 [00:01<06:35,  1.68s/ba]
  1%|          | 2/236 [00:03<07:17,  1.87s/ba]
  1%|▏         | 3/236 [00:05<06:56,  1.79s/ba]
  2%|▏         | 4/236 [00:06<06:22,  1.65s/ba]
  2%|▏         | 5/236 [00:08<06:02,  1.57s/ba]
  3%|▎         | 6/236 [00:09<05:36,  1.46s/ba]
  3%|▎         | 7/236 [00:11<05:50,  1.53s/ba]
  3%|▎         | 8/236 [00:12<05:55,  1.56s/ba]
  4%|▍         | 9/236 [00:14<05:40,  1.50s/ba]
  4%|▍         | 10/236 [00:15<05:43,  1.52s/ba]
  5%|▍         | 11/236 [00:17<05:33,  1.48s/ba]
  5%|▌         | 12/236 [00:18<05:32,  1.49s/ba]
  6%|▌         | 13/236 [00:20<05:37,  1.51s/ba]
  6%|▌         | 14/236 [00:21<05:15,  1.42s/ba]
  6%|▋         | 15/236 [00:22<05:23,  1.47s/ba]
  7%|▋         | 16/236 [00:24<05:07,  1.40s/ba]
  7%|▋         | 17/236 [00:25<05:03,  1.39s/ba]
  8%|▊         | 18/236 [00:27<05:08,  1.42s/ba]
  8%|▊         | 19/236 [00:28<05:06,  1.41s/ba]
  8%|▊         | 20/236 [00:30<05:22,  1.50s/ba]
  9%|▉         | 21/236 [00:32<05:53,  1.64s/ba]
  9%|▉         | 22/236 [00:33<05:56,  1.67s/ba]
 10%|▉         | 23/236 [00:35<05:40,  1.60s/ba]
 10%|█         | 24/236 [00:36<05:41,  1.61s/ba]
 11%|█         | 25/236 [00:38<05:28,  1.56s/ba]
 11%|█         | 26/236 [00:40<05:36,  1.60s/ba]
 11%|█▏        | 27/236 [00:41<05:47,  1.66s/ba]
 12%|█▏        | 28/236 [00:43<05:33,  1.60s/ba]
 12%|█▏        | 29/236 [00:45<06:03,  1.76s/ba]
 13%|█▎        | 30/236 [00:47<06:04,  1.77s/ba]
 13%|█▎        | 31/236 [00:48<05:42,  1.67s/ba]
 14%|█▎        | 32/236 [00:50<05:21,  1.58s/ba]
 14%|█▍        | 33/236 [00:51<05:22,  1.59s/ba]
 14%|█▍        | 34/236 [00:53<05:07,  1.52s/ba]
 15%|█▍        | 35/236 [00:54<05:07,  1.53s/ba]
 15%|█▌        | 36/236 [00:56<05:06,  1.53s/ba]
 16%|█▌        | 37/236 [00:57<04:51,  1.47s/ba]
 16%|█▌        | 38/236 [00:59<04:57,  1.50s/ba]
 17%|█▋        | 39/236 [01:00<05:07,  1.56s/ba]
 17%|█▋        | 40/236 [01:02<05:10,  1.59s/ba]
 17%|█▋        | 41/236 [01:04<05:15,  1.62s/ba]
 18%|█▊        | 42/236 [01:05<04:43,  1.46s/ba]
 18%|█▊        | 43/236 [01:06<04:48,  1.49s/ba]
 19%|█▊        | 44/236 [01:08<04:44,  1.48s/ba]
 19%|█▉        | 45/236 [01:09<04:43,  1.48s/ba]
 19%|█▉        | 46/236 [01:11<05:25,  1.71s/ba]
 20%|█▉        | 47/236 [01:14<05:59,  1.90s/ba]
 20%|██        | 48/236 [01:16<06:18,  2.01s/ba]
 21%|██        | 49/236 [01:18<06:03,  1.94s/ba]
 21%|██        | 50/236 [01:19<05:27,  1.76s/ba]
 22%|██▏       | 51/236 [01:21<05:20,  1.73s/ba]
 22%|██▏       | 52/236 [01:22<05:11,  1.70s/ba]
 22%|██▏       | 53/236 [01:24<04:49,  1.58s/ba]
 23%|██▎       | 54/236 [01:25<04:41,  1.55s/ba]
 23%|██▎       | 55/236 [01:26<04:25,  1.47s/ba]
 24%|██▎       | 56/236 [01:28<04:28,  1.49s/ba]
 24%|██▍       | 57/236 [01:30<04:29,  1.51s/ba]
 25%|██▍       | 58/236 [01:31<04:27,  1.50s/ba]
 25%|██▌       | 59/236 [01:33<04:32,  1.54s/ba]
 25%|██▌       | 60/236 [01:34<04:09,  1.42s/ba]
 26%|██▌       | 61/236 [01:35<04:16,  1.47s/ba]
 26%|██▋       | 62/236 [01:37<04:11,  1.44s/ba]
 27%|██▋       | 63/236 [01:38<04:12,  1.46s/ba]
 27%|██▋       | 64/236 [01:40<04:21,  1.52s/ba]
 28%|██▊       | 65/236 [01:42<04:31,  1.59s/ba]
 28%|██▊       | 66/236 [01:43<04:18,  1.52s/ba]
 28%|██▊       | 67/236 [01:45<04:19,  1.54s/ba]
 29%|██▉       | 68/236 [01:47<04:48,  1.72s/ba]
 29%|██▉       | 69/236 [01:48<04:32,  1.63s/ba]
 30%|██▉       | 70/236 [01:50<04:31,  1.63s/ba]
 30%|███       | 71/236 [01:51<04:15,  1.55s/ba]
 31%|███       | 72/236 [01:53<04:16,  1.56s/ba]
 31%|███       | 73/236 [01:54<04:09,  1.53s/ba]
 31%|███▏      | 74/236 [01:56<04:04,  1.51s/ba]
 32%|███▏      | 75/236 [01:57<04:14,  1.58s/ba]
 32%|███▏      | 76/236 [02:00<04:44,  1.78s/ba]
 33%|███▎      | 77/236 [02:02<04:46,  1.80s/ba]
 33%|███▎      | 78/236 [02:03<04:27,  1.70s/ba]
 33%|███▎      | 79/236 [02:05<04:22,  1.67s/ba]
 34%|███▍      | 80/236 [02:06<04:16,  1.65s/ba]
 34%|███▍      | 81/236 [02:08<04:06,  1.59s/ba]
 35%|███▍      | 82/236 [02:09<04:03,  1.58s/ba]
 35%|███▌      | 83/236 [02:11<03:59,  1.57s/ba]
 36%|███▌      | 84/236 [02:12<04:05,  1.62s/ba]
 36%|███▌      | 85/236 [02:14<04:18,  1.71s/ba]
 36%|███▋      | 86/236 [02:16<04:03,  1.62s/ba]
 37%|███▋      | 87/236 [02:17<04:00,  1.61s/ba]
 37%|███▋      | 88/236 [02:19<04:02,  1.64s/ba]
 38%|███▊      | 89/236 [02:21<03:54,  1.60s/ba]
 38%|███▊      | 90/236 [02:22<03:53,  1.60s/ba]
 39%|███▊      | 91/236 [02:24<04:00,  1.66s/ba]
 39%|███▉      | 92/236 [02:26<03:54,  1.63s/ba]
 39%|███▉      | 93/236 [02:27<04:00,  1.68s/ba]
 40%|███▉      | 94/236 [02:30<04:25,  1.87s/ba]
 40%|████      | 95/236 [02:32<04:28,  1.90s/ba]
 41%|████      | 96/236 [02:33<04:17,  1.84s/ba]
 41%|████      | 97/236 [02:35<04:08,  1.79s/ba]
 42%|████▏     | 98/236 [02:37<04:15,  1.85s/ba]
 42%|████▏     | 99/236 [02:39<04:12,  1.85s/ba]
 42%|████▏     | 100/236 [02:40<03:57,  1.75s/ba]
 43%|████▎     | 101/236 [02:42<03:56,  1.75s/ba]
 43%|████▎     | 102/236 [02:44<03:54,  1.75s/ba]
 44%|████▎     | 103/236 [02:45<03:40,  1.66s/ba]
 44%|████▍     | 104/236 [02:47<03:33,  1.61s/ba]
 44%|████▍     | 105/236 [02:48<03:24,  1.56s/ba]
 45%|████▍     | 106/236 [02:50<03:15,  1.50s/ba]
 45%|████▌     | 107/236 [02:51<03:17,  1.53s/ba]
 46%|████▌     | 108/236 [02:53<03:11,  1.50s/ba]
 46%|████▌     | 109/236 [02:54<03:11,  1.51s/ba]
 47%|████▋     | 110/236 [02:56<03:06,  1.48s/ba]
 47%|████▋     | 111/236 [02:58<03:21,  1.61s/ba]
 47%|████▋     | 112/236 [02:59<03:18,  1.60s/ba]
 48%|████▊     | 113/236 [03:01<03:15,  1.59s/ba]
 48%|████▊     | 114/236 [03:02<03:10,  1.56s/ba]
 49%|████▊     | 115/236 [03:04<03:14,  1.61s/ba]
 49%|████▉     | 116/236 [03:06<03:13,  1.62s/ba]
 50%|████▉     | 117/236 [03:07<03:05,  1.56s/ba]
 50%|█████     | 118/236 [03:09<03:08,  1.60s/ba]
 50%|█████     | 119/236 [03:10<03:05,  1.58s/ba]
 51%|█████     | 120/236 [03:12<03:02,  1.58s/ba]
 51%|█████▏    | 121/236 [03:13<03:05,  1.61s/ba]
 52%|█████▏    | 122/236 [03:15<03:02,  1.60s/ba]
 52%|█████▏    | 123/236 [03:16<02:52,  1.52s/ba]
 53%|█████▎    | 124/236 [03:18<02:48,  1.51s/ba]
 53%|█████▎    | 125/236 [03:19<02:46,  1.50s/ba]
 53%|█████▎    | 126/236 [03:21<02:49,  1.54s/ba]
 54%|█████▍    | 127/236 [03:22<02:45,  1.52s/ba]
 54%|█████▍    | 128/236 [03:24<02:41,  1.50s/ba]
 55%|█████▍    | 129/236 [03:26<02:56,  1.65s/ba]
 55%|█████▌    | 130/236 [03:28<02:56,  1.67s/ba]
 56%|█████▌    | 131/236 [03:29<02:52,  1.64s/ba]
 56%|█████▌    | 132/236 [03:31<02:55,  1.69s/ba]
 56%|█████▋    | 133/236 [03:32<02:45,  1.60s/ba]
 57%|█████▋    | 134/236 [03:34<02:40,  1.58s/ba]
 57%|█████▋    | 135/236 [03:36<02:46,  1.65s/ba]
 58%|█████▊    | 136/236 [03:37<02:38,  1.58s/ba]
 58%|█████▊    | 137/236 [03:39<02:32,  1.54s/ba]
 58%|█████▊    | 138/236 [03:40<02:42,  1.65s/ba]
 59%|█████▉    | 139/236 [03:42<02:37,  1.62s/ba]
 59%|█████▉    | 140/236 [03:44<02:34,  1.61s/ba]
 60%|█████▉    | 141/236 [03:45<02:34,  1.62s/ba]
 60%|██████    | 142/236 [03:47<02:25,  1.54s/ba]
 61%|██████    | 143/236 [03:48<02:19,  1.50s/ba]
 61%|██████    | 144/236 [03:50<02:37,  1.72s/ba]
 61%|██████▏   | 145/236 [03:52<02:35,  1.71s/ba]
 62%|██████▏   | 146/236 [03:53<02:27,  1.64s/ba]
 62%|██████▏   | 147/236 [03:56<02:42,  1.83s/ba]
 63%|██████▎   | 148/236 [03:57<02:39,  1.81s/ba]
 63%|██████▎   | 149/236 [03:59<02:18,  1.60s/ba]
 64%|██████▎   | 150/236 [04:00<02:13,  1.55s/ba]
 64%|██████▍   | 151/236 [04:02<02:10,  1.54s/ba]
 64%|██████▍   | 152/236 [04:03<02:07,  1.52s/ba]
 65%|██████▍   | 153/236 [04:05<02:13,  1.61s/ba]
 65%|██████▌   | 154/236 [04:06<02:13,  1.63s/ba]
 66%|██████▌   | 155/236 [04:08<02:08,  1.59s/ba]
 66%|██████▌   | 156/236 [04:10<02:18,  1.73s/ba]
 67%|██████▋   | 157/236 [04:12<02:21,  1.79s/ba]
 67%|██████▋   | 158/236 [04:14<02:21,  1.81s/ba]
 67%|██████▋   | 159/236 [04:15<02:12,  1.72s/ba]
 68%|██████▊   | 160/236 [04:17<02:09,  1.70s/ba]
 68%|██████▊   | 161/236 [04:19<02:07,  1.71s/ba]
 69%|██████▊   | 162/236 [04:20<02:05,  1.69s/ba]
 69%|██████▉   | 163/236 [04:22<02:01,  1.66s/ba]
 69%|██████▉   | 164/236 [04:24<02:04,  1.73s/ba]
 70%|██████▉   | 165/236 [04:26<02:05,  1.77s/ba]
 70%|███████   | 166/236 [04:27<02:00,  1.72s/ba]
 71%|███████   | 167/236 [04:28<01:47,  1.56s/ba]
 71%|███████   | 168/236 [04:30<01:44,  1.54s/ba]
 72%|███████▏  | 169/236 [04:32<01:46,  1.60s/ba]
 72%|███████▏  | 170/236 [04:33<01:48,  1.64s/ba]
 72%|███████▏  | 171/236 [04:35<01:43,  1.60s/ba]
 73%|███████▎  | 172/236 [04:37<01:46,  1.66s/ba]
 73%|███████▎  | 173/236 [04:38<01:44,  1.66s/ba]
 74%|███████▎  | 174/236 [04:40<01:38,  1.59s/ba]
 74%|███████▍  | 175/236 [04:41<01:37,  1.59s/ba]
 75%|███████▍  | 176/236 [04:43<01:35,  1.60s/ba]
 75%|███████▌  | 177/236 [04:44<01:30,  1.54s/ba]
 75%|███████▌  | 178/236 [04:46<01:29,  1.55s/ba]
 76%|███████▌  | 179/236 [04:47<01:25,  1.49s/ba]
 76%|███████▋  | 180/236 [04:49<01:25,  1.53s/ba]
 77%|███████▋  | 181/236 [04:51<01:28,  1.60s/ba]
 77%|███████▋  | 182/236 [04:53<01:38,  1.83s/ba]
 78%|███████▊  | 183/236 [04:55<01:32,  1.75s/ba]
 78%|███████▊  | 184/236 [04:57<01:38,  1.89s/ba]
 78%|███████▊  | 185/236 [04:59<01:33,  1.84s/ba]
 79%|███████▉  | 186/236 [05:00<01:25,  1.72s/ba]
 79%|███████▉  | 187/236 [05:01<01:18,  1.60s/ba]
 80%|███████▉  | 188/236 [05:03<01:17,  1.60s/ba]
 80%|████████  | 189/236 [05:04<01:13,  1.56s/ba]
 81%|████████  | 190/236 [05:06<01:11,  1.55s/ba]
 81%|████████  | 191/236 [05:08<01:10,  1.56s/ba]
 81%|████████▏ | 192/236 [05:09<01:08,  1.56s/ba]
 82%|████████▏ | 193/236 [05:11<01:08,  1.60s/ba]
 82%|████████▏ | 194/236 [05:12<01:07,  1.62s/ba]
 83%|████████▎ | 195/236 [05:14<01:04,  1.58s/ba]
 83%|████████▎ | 196/236 [05:16<01:02,  1.57s/ba]
 83%|████████▎ | 197/236 [05:17<01:04,  1.65s/ba]
 84%|████████▍ | 198/236 [05:19<01:07,  1.78s/ba]
 84%|████████▍ | 199/236 [05:21<01:01,  1.68s/ba]
 85%|████████▍ | 200/236 [05:23<01:05,  1.81s/ba]
 85%|████████▌ | 201/236 [05:25<01:00,  1.73s/ba]
 86%|████████▌ | 202/236 [05:26<00:58,  1.73s/ba]
 86%|████████▌ | 203/236 [05:28<00:56,  1.70s/ba]
 86%|████████▋ | 204/236 [05:29<00:51,  1.61s/ba]
 87%|████████▋ | 205/236 [05:31<00:53,  1.73s/ba]
 87%|████████▋ | 206/236 [05:33<00:51,  1.70s/ba]
 88%|████████▊ | 207/236 [05:34<00:47,  1.65s/ba]
 88%|████████▊ | 208/236 [05:36<00:43,  1.56s/ba]
 89%|████████▊ | 209/236 [05:38<00:45,  1.68s/ba]
 89%|████████▉ | 210/236 [05:39<00:43,  1.67s/ba]
 89%|████████▉ | 211/236 [05:41<00:40,  1.63s/ba]
 90%|████████▉ | 212/236 [05:43<00:38,  1.61s/ba]
 90%|█████████ | 213/236 [05:44<00:35,  1.56s/ba]
 91%|█████████ | 214/236 [05:45<00:32,  1.48s/ba]
 91%|█████████ | 215/236 [05:47<00:31,  1.48s/ba]
 92%|█████████▏| 216/236 [05:48<00:29,  1.46s/ba]
 92%|█████████▏| 217/236 [05:50<00:28,  1.50s/ba]
 92%|█████████▏| 218/236 [05:51<00:27,  1.53s/ba]
 93%|█████████▎| 219/236 [05:54<00:29,  1.74s/ba]
 93%|█████████▎| 220/236 [05:55<00:26,  1.65s/ba]
 94%|█████████▎| 221/236 [05:57<00:24,  1.63s/ba]
 94%|█████████▍| 222/236 [05:58<00:22,  1.59s/ba]
 94%|█████████▍| 223/236 [06:00<00:20,  1.58s/ba]
 95%|█████████▍| 224/236 [06:01<00:19,  1.58s/ba]
 95%|█████████▌| 225/236 [06:03<00:17,  1.55s/ba]
 96%|█████████▌| 226/236 [06:05<00:17,  1.77s/ba]
 96%|█████████▌| 227/236 [06:07<00:16,  1.82s/ba]
 97%|█████████▋| 228/236 [06:09<00:14,  1.76s/ba]
 97%|█████████▋| 229/236 [06:10<00:11,  1.66s/ba]
 97%|█████████▋| 230/236 [06:12<00:09,  1.65s/ba]
 98%|█████████▊| 231/236 [06:13<00:07,  1.58s/ba]
 98%|█████████▊| 232/236 [06:14<00:06,  1.51s/ba]
 99%|█████████▊| 233/236 [06:16<00:04,  1.54s/ba]
 99%|█████████▉| 234/236 [06:17<00:02,  1.48s/ba]
100%|█████████▉| 235/236 [06:19<00:01,  1.66s/ba]
100%|██████████| 236/236 [06:21<00:00,  1.58s/ba]
100%|██████████| 236/236 [06:21<00:00,  1.62s/ba]

  0%|          | 0/121 [00:00<?, ?ba/s]12/20/2022 10:18:52 - INFO - datasets.arrow_dataset - Caching processed dataset at /home/milan/.cache/huggingface/datasets/mozilla-foundation___common_voice_11_0/cs/11.0.0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f/cache-ac05cdeebe77a259.arrow

  1%|          | 1/121 [00:01<02:56,  1.47s/ba]
  2%|▏         | 2/121 [00:03<03:08,  1.59s/ba]
  2%|▏         | 3/121 [00:04<02:53,  1.47s/ba]
  3%|▎         | 4/121 [00:06<03:07,  1.60s/ba]
  4%|▍         | 5/121 [00:08<03:36,  1.87s/ba]
  5%|▍         | 6/121 [00:10<03:27,  1.80s/ba]
  6%|▌         | 7/121 [00:11<03:18,  1.74s/ba]
  7%|▋         | 8/121 [00:13<03:17,  1.75s/ba]
  7%|▋         | 9/121 [00:15<03:11,  1.71s/ba]
  8%|▊         | 10/121 [00:16<03:07,  1.69s/ba]
  9%|▉         | 11/121 [00:18<03:04,  1.68s/ba]
 10%|▉         | 12/121 [00:20<03:09,  1.74s/ba]
 11%|█         | 13/121 [00:21<02:59,  1.66s/ba]
 12%|█▏        | 14/121 [00:23<02:52,  1.61s/ba]
 12%|█▏        | 15/121 [00:25<02:53,  1.64s/ba]
 13%|█▎        | 16/121 [00:27<03:08,  1.80s/ba]
 14%|█▍        | 17/121 [00:28<03:01,  1.75s/ba]
 15%|█▍        | 18/121 [00:30<02:52,  1.67s/ba]
 16%|█▌        | 19/121 [00:32<02:49,  1.66s/ba]
 17%|█▋        | 20/121 [00:33<02:42,  1.61s/ba]
 17%|█▋        | 21/121 [00:35<02:41,  1.62s/ba]
 18%|█▊        | 22/121 [00:36<02:32,  1.54s/ba]
 19%|█▉        | 23/121 [00:37<02:26,  1.50s/ba]
 20%|█▉        | 24/121 [00:39<02:30,  1.56s/ba]
 21%|██        | 25/121 [00:41<02:28,  1.54s/ba]
 21%|██▏       | 26/121 [00:42<02:34,  1.62s/ba]
 22%|██▏       | 27/121 [00:44<02:33,  1.63s/ba]
 23%|██▎       | 28/121 [00:46<02:25,  1.57s/ba]
 24%|██▍       | 29/121 [00:47<02:23,  1.56s/ba]
 25%|██▍       | 30/121 [00:49<02:22,  1.57s/ba]
 26%|██▌       | 31/121 [00:50<02:16,  1.52s/ba]
 26%|██▋       | 32/121 [00:52<02:21,  1.59s/ba]
 27%|██▋       | 33/121 [00:53<02:16,  1.55s/ba]
 28%|██▊       | 34/121 [00:55<02:12,  1.52s/ba]
 29%|██▉       | 35/121 [00:57<02:18,  1.61s/ba]
 30%|██▉       | 36/121 [00:58<02:17,  1.61s/ba]
 31%|███       | 37/121 [00:59<02:06,  1.51s/ba]
 31%|███▏      | 38/121 [01:01<02:06,  1.52s/ba]
 32%|███▏      | 39/121 [01:02<01:56,  1.42s/ba]
 33%|███▎      | 40/121 [01:04<01:59,  1.47s/ba]
 34%|███▍      | 41/121 [01:05<01:56,  1.46s/ba]
 35%|███▍      | 42/121 [01:07<01:53,  1.44s/ba]
 36%|███▌      | 43/121 [01:08<01:56,  1.50s/ba]
 36%|███▋      | 44/121 [01:10<01:54,  1.48s/ba]
 37%|███▋      | 45/121 [01:11<01:56,  1.53s/ba]
 38%|███▊      | 46/121 [01:13<01:56,  1.55s/ba]
 39%|███▉      | 47/121 [01:14<01:51,  1.51s/ba]
 40%|███▉      | 48/121 [01:16<01:52,  1.55s/ba]
 40%|████      | 49/121 [01:18<01:52,  1.57s/ba]
 41%|████▏     | 50/121 [01:19<01:49,  1.54s/ba]
 42%|████▏     | 51/121 [01:21<01:48,  1.55s/ba]
 43%|████▎     | 52/121 [01:22<01:46,  1.54s/ba]
 44%|████▍     | 53/121 [01:23<01:34,  1.39s/ba]
 45%|████▍     | 54/121 [01:25<01:42,  1.54s/ba]
 45%|████▌     | 55/121 [01:27<01:45,  1.60s/ba]
 46%|████▋     | 56/121 [01:28<01:40,  1.54s/ba]
 47%|████▋     | 57/121 [01:30<01:38,  1.55s/ba]
 48%|████▊     | 58/121 [01:31<01:34,  1.51s/ba]
 49%|████▉     | 59/121 [01:33<01:34,  1.53s/ba]
 50%|████▉     | 60/121 [01:34<01:35,  1.56s/ba]
 50%|█████     | 61/121 [01:36<01:28,  1.47s/ba]
 51%|█████     | 62/121 [01:38<01:36,  1.64s/ba]
 52%|█████▏    | 63/121 [01:39<01:35,  1.64s/ba]
 53%|█████▎    | 64/121 [01:41<01:37,  1.71s/ba]
 54%|█████▎    | 65/121 [01:43<01:29,  1.60s/ba]
 55%|█████▍    | 66/121 [01:44<01:25,  1.55s/ba]
 55%|█████▌    | 67/121 [01:45<01:19,  1.48s/ba]
 56%|█████▌    | 68/121 [01:47<01:18,  1.48s/ba]
 57%|█████▋    | 69/121 [01:48<01:16,  1.48s/ba]
 58%|█████▊    | 70/121 [01:50<01:17,  1.52s/ba]
 59%|█████▊    | 71/121 [01:51<01:14,  1.50s/ba]
 60%|█████▉    | 72/121 [01:53<01:14,  1.53s/ba]
 60%|██████    | 73/121 [01:55<01:19,  1.65s/ba]
 61%|██████    | 74/121 [01:57<01:18,  1.67s/ba]
 62%|██████▏   | 75/121 [01:58<01:11,  1.56s/ba]
 63%|██████▎   | 76/121 [02:00<01:19,  1.76s/ba]
 64%|██████▎   | 77/121 [02:02<01:14,  1.68s/ba]
 64%|██████▍   | 78/121 [02:03<01:07,  1.58s/ba]
 65%|██████▌   | 79/121 [02:05<01:08,  1.62s/ba]
 66%|██████▌   | 80/121 [02:06<01:07,  1.64s/ba]
 67%|██████▋   | 81/121 [02:08<01:05,  1.65s/ba]
 68%|██████▊   | 82/121 [02:09<01:02,  1.59s/ba]
 69%|██████▊   | 83/121 [02:11<01:02,  1.63s/ba]
 69%|██████▉   | 84/121 [02:13<01:00,  1.63s/ba]
 70%|███████   | 85/121 [02:14<00:57,  1.61s/ba]
 71%|███████   | 86/121 [02:16<01:01,  1.76s/ba]
 72%|███████▏  | 87/121 [02:18<00:56,  1.67s/ba]
 73%|███████▎  | 88/121 [02:19<00:50,  1.53s/ba]
 74%|███████▎  | 89/121 [02:21<00:51,  1.62s/ba]
 74%|███████▍  | 90/121 [02:23<00:51,  1.67s/ba]
 75%|███████▌  | 91/121 [02:24<00:47,  1.59s/ba]
 76%|███████▌  | 92/121 [02:26<00:47,  1.63s/ba]
 77%|███████▋  | 93/121 [02:27<00:42,  1.51s/ba]
 78%|███████▊  | 94/121 [02:29<00:40,  1.51s/ba]
 79%|███████▊  | 95/121 [02:30<00:38,  1.49s/ba]
 79%|███████▉  | 96/121 [02:31<00:36,  1.46s/ba]
 80%|████████  | 97/121 [02:33<00:35,  1.49s/ba]
 81%|████████  | 98/121 [02:34<00:33,  1.45s/ba]
 82%|████████▏ | 99/121 [02:36<00:33,  1.53s/ba]
 83%|████████▎ | 100/121 [02:38<00:32,  1.57s/ba]
 83%|████████▎ | 101/121 [02:39<00:28,  1.43s/ba]
 84%|████████▍ | 102/121 [02:41<00:28,  1.50s/ba]
 85%|████████▌ | 103/121 [02:42<00:27,  1.51s/ba]
 86%|████████▌ | 104/121 [02:44<00:25,  1.50s/ba]
 87%|████████▋ | 105/121 [02:46<00:26,  1.67s/ba]
 88%|████████▊ | 106/121 [02:47<00:25,  1.70s/ba]
 88%|████████▊ | 107/121 [02:49<00:22,  1.59s/ba]
 89%|████████▉ | 108/121 [02:51<00:21,  1.65s/ba]
 90%|█████████ | 109/121 [02:52<00:20,  1.67s/ba]
 91%|█████████ | 110/121 [02:53<00:17,  1.55s/ba]
 92%|█████████▏| 111/121 [02:55<00:15,  1.55s/ba]
 93%|█████████▎| 112/121 [02:56<00:13,  1.52s/ba]
 93%|█████████▎| 113/121 [02:58<00:12,  1.59s/ba]
 94%|█████████▍| 114/121 [03:00<00:11,  1.69s/ba]
 95%|█████████▌| 115/121 [03:01<00:08,  1.49s/ba]
 96%|█████████▌| 116/121 [03:03<00:07,  1.59s/ba]
 97%|█████████▋| 117/121 [03:05<00:06,  1.67s/ba]
 98%|█████████▊| 118/121 [03:07<00:05,  1.76s/ba]
 98%|█████████▊| 119/121 [03:08<00:03,  1.66s/ba]
 99%|█████████▉| 120/121 [03:10<00:01,  1.67s/ba]
100%|██████████| 121/121 [03:11<00:00,  1.40s/ba]
100%|██████████| 121/121 [03:11<00:00,  1.58s/ba]
[INFO|feature_extraction_utils.py:368] 2022-12-20 10:22:02,746 >> Feature extractor saved in ./preprocessor_config.json
[INFO|tokenization_utils_base.py:2157] 2022-12-20 10:22:02,747 >> tokenizer config file saved in ./tokenizer_config.json
[INFO|tokenization_utils_base.py:2164] 2022-12-20 10:22:02,747 >> Special tokens file saved in ./special_tokens_map.json
[INFO|tokenization_utils_base.py:2210] 2022-12-20 10:22:02,747 >> added tokens file saved in ./added_tokens.json
[INFO|configuration_utils.py:447] 2022-12-20 10:22:02,828 >> Configuration saved in ./config.json
[INFO|image_processing_utils.py:294] 2022-12-20 10:22:02,830 >> loading configuration file ./preprocessor_config.json
[INFO|feature_extraction_utils.py:462] 2022-12-20 10:22:02,835 >> loading configuration file ./preprocessor_config.json
[INFO|feature_extraction_utils.py:501] 2022-12-20 10:22:02,850 >> Feature extractor WhisperFeatureExtractor {
  "chunk_length": 30,
  "feature_extractor_type": "WhisperFeatureExtractor",
  "feature_size": 80,
  "hop_length": 160,
  "mel_filters": [
    [
      -0.0,
      0.02486259490251541,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.001990821911022067,
      0.022871771827340126,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.003981643822044134,
      0.02088095061480999,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0059724655002355576,
      0.018890129402279854,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.007963287644088268,
      0.01689930632710457,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.009954108856618404,
      0.014908484183251858,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.011944931000471115,
      0.012917662039399147,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.013935752213001251,
      0.010926840826869011,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.015926575288176537,
      0.0089360186830163,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.017917396500706673,
      0.006945197004824877,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01990821771323681,
      0.004954374860972166,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.021899040788412094,
      0.0029635531827807426,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.02388986200094223,
      0.0009727313299663365,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.025880683213472366,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.025835324078798294,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0010180906392633915,
      0.023844502866268158,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.003008912317454815,
      0.021853681653738022,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.004999734461307526,
      0.019862858578562737,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.006990555673837662,
      0.0178720373660326,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.008981377817690372,
      0.015881216153502464,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.010972199961543083,
      0.013890394009649754,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01296302117407322,
      0.011899571865797043,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01495384331792593,
      0.009908749721944332,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01694466546177864,
      0.007917927578091621,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.018935488536953926,
      0.005927106365561485,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.020874010398983955,
      0.004040425643324852,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.022114217281341553,
      0.0033186059445142746,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.02173672430217266,
      0.0036109676584601402,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.020497702062129974,
      0.004762193653732538,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.018486659973859787,
      0.006592618301510811,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.01585603691637516,
      0.00896277092397213,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.012738768011331558,
      0.011751330457627773,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.009250369854271412,
      0.014853144995868206,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.005490840878337622,
      0.018177473917603493,
      0.0028155462350696325,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0015463664894923568,
      0.01632951945066452,
      0.007420188747346401,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.011181050911545753,
      0.012018864043056965,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.006065350491553545,
      0.016561277210712433,
      0.004360878840088844,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0010297985281795263,
      0.012770536355674267,
      0.009707189165055752,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.006986402906477451,
      0.01485429983586073,
      0.004391219466924667,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.001418047584593296,
      0.011486922390758991,
      0.010089744813740253,
      0.00040022286702878773,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.005411104764789343,
      0.014735566452145576,
      0.006518189795315266,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00827841367572546,
      0.012277561239898205,
      0.00396781275048852,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.002187808509916067,
      0.010184479877352715,
      0.00998187530785799,
      0.0022864851634949446,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00386943481862545,
      0.011274894699454308,
      0.008466221392154694,
      0.0013397691072896123,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.004820294212549925,
      0.011678251437842846,
      0.007608682848513126,
      0.0010091039584949613,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.005156961735337973,
      0.011507894843816757,
      0.007301822770386934,
      0.0011901655234396458,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.004982104524970055,
      0.010863498784601688,
      0.007451189681887627,
      0.001791381393559277,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.004385921638458967,
      0.009832492098212242,
      0.007973956875503063,
      0.002732589840888977,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0034474546555429697,
      0.008491347543895245,
      0.008797688409686089,
      0.00394382793456316,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0022357646375894547,
      0.0069067515432834625,
      0.009859241545200348,
      0.005364237818866968,
      0.0008692338014952838,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0008110002381727099,
      0.005136650986969471,
      0.00946230161935091,
      0.0069410777650773525,
      0.0027783995028585196,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.003231203882023692,
      0.007237049750983715,
      0.00862883497029543,
      0.004773912951350212,
      0.0009189908159896731,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.001233637798577547,
      0.0049433219246566296,
      0.008653006516397,
      0.006818502210080624,
      0.003248583758249879,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0026164355222135782,
      0.006051854696124792,
      0.008880467154085636,
      0.005574479699134827,
      0.002268492942675948,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0002863667905330658,
      0.003467798000201583,
      0.0066492292098701,
      0.00787146482616663,
      0.004809896927326918,
      0.0017483289120718837,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0009245910914614797,
      0.0038708120118826628,
      0.00681703258305788,
      0.007283343467861414,
      0.004448124207556248,
      0.0016129047144204378,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0011703289346769452,
      0.003898728871718049,
      0.006627128925174475,
      0.0070473202504217625,
      0.004421714693307877,
      0.0017961094854399562,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0010892992140725255,
      0.003615982597693801,
      0.006142666097730398,
      0.007102936040610075,
      0.004671447444707155,
      0.002239959081634879,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0007392280967906117,
      0.0030791081953793764,
      0.005418988410383463,
      0.007397185545414686,
      0.005145462695509195,
      0.002893739379942417,
      0.0006420162972062826,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00017068670422304422,
      0.0023375742603093386,
      0.004504461772739887,
      0.0066713495180010796,
      0.005798479542136192,
      0.003713231300935149,
      0.0016279831761494279,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0014345343224704266,
      0.0034412189852446318,
      0.005447904113680124,
      0.006591092795133591,
      0.004660011734813452,
      0.002728930441662669,
      0.0007978491485118866,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0004075043834745884,
      0.002265830524265766,
      0.004124156199395657,
      0.005982482805848122,
      0.005700822453945875,
      0.003912510350346565,
      0.0021241982467472553,
      0.0003358862304594368,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0010099108330905437,
      0.002730846870690584,
      0.004451782442629337,
      0.006172718480229378,
      0.005150905344635248,
      0.0034948070533573627,
      0.0018387088784947991,
      0.0001826105872169137,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0012943691108375788,
      0.002888072282075882,
      0.004481775686144829,
      0.006075479090213776,
      0.0048866597935557365,
      0.003353001084178686,
      0.0018193417927250266,
      0.00028568264679051936,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0013131388695910573,
      0.0027890161145478487,
      0.004264893010258675,
      0.0057407706044614315,
      0.004859979264438152,
      0.0034397069830447435,
      0.0020194342359900475,
      0.0005991620710119605,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0011121684219688177,
      0.002478930866345763,
      0.0038456933107227087,
      0.0052124555222690105,
      0.005028639920055866,
      0.0037133716978132725,
      0.002398103242740035,
      0.0010828346712514758,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0007317548734135926,
      0.0019974694587290287,
      0.003263183869421482,
      0.004528898745775223,
      0.005355686880648136,
      0.004137659445405006,
      0.0029196315445005894,
      0.0017016039928421378,
      0.0004835762665607035,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00020713974663522094,
      0.0013792773243039846,
      0.0025514145381748676,
      0.003723552217707038,
      0.004895689897239208,
      0.004680895246565342,
      0.0035529187880456448,
      0.0024249425623565912,
      0.0012969663366675377,
      0.00016899015463422984,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0006545265205204487,
      0.0017400053329765797,
      0.0028254841454327106,
      0.003910962492227554,
      0.004996441304683685,
      0.0042709787376224995,
      0.003226396394893527,
      0.002181813819333911,
      0.0011372314766049385,
      9.264905384043232e-05,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.000854626705404371,
      0.001859853626228869,
      0.002865080488845706,
      0.003870307235047221,
      0.00487553421407938,
      0.00408313749358058,
      0.003115783678367734,
      0.0021484296303242445,
      0.001181075582280755,
      0.0002137213887181133,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0008483415003865957,
      0.0017792496364563704,
      0.0027101580053567886,
      0.0036410661414265633,
      0.004571974277496338,
      0.004079728852957487,
      0.003183893393725157,
      0.002288057701662183,
      0.0013922222424298525,
      0.0004963868414051831,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0006716204807162285,
      0.0015337044605985284,
      0.002395788673311472,
      0.0032578727696090937,
      0.004119956865906715,
      0.004227725323289633,
      0.0033981208689510822,
      0.0025685166474431753,
      0.0017389123095199466,
      0.0009093079133890569,
      7.970355363795534e-05,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0003559796023182571,
      0.0011543278815224767,
      0.0019526762189343572,
      0.002751024439930916,
      0.0035493727773427963,
      0.004347721114754677,
      0.0037299629766494036,
      0.002961693098768592,
      0.00219342322088778,
      0.0014251532265916467,
      0.0006568834069184959,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0006682946113869548,
      0.0014076193328946829,
      0.0021469437051564455,
      0.002886268775910139,
      0.0036255933810025454,
      0.004154576454311609,
      0.0034431067761033773,
      0.0027316368650645018,
      0.0020201667211949825,
      0.0013086966937407851,
      0.0005972267827019095,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      9.926508937496692e-05,
      0.0007839298341423273,
      0.001468594535253942,
      0.0021532592363655567,
      0.0028379240538924932,
      0.0035225888714194298,
      0.0039915177039802074,
      0.0033326479606330395,
      0.002673778682947159,
      0.002014909405261278,
      0.0013560398947447538,
      0.0006971705006435513,
      3.8301113818306476e-05,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.00010181095422012731,
      0.0007358568836934865,
      0.0013699028640985489,
      0.0020039486698806286,
      0.002637994708493352,
      0.0032720407471060753,
      0.003906086552888155,
      0.0033682563807815313,
      0.0027580985333770514,
      0.002147940918803215,
      0.0015377833042293787,
      0.0009276255150325596,
      0.000317467754939571,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0005530364578589797,
      0.0011402058880776167,
      0.0017273754347115755,
      0.0023145449813455343,
      0.002901714527979493,
      0.003488884074613452,
      0.003523340215906501,
      0.002958292607218027,
      0.002393245231360197,
      0.0018281979719176888,
      0.001263150479644537,
      0.0006981031037867069,
      0.0001330557424807921,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0002608386566862464,
      0.0008045974536798894,
      0.0013483562506735325,
      0.0018921148730441928,
      0.0024358737282454967,
      0.002979632467031479,
      0.003523391205817461,
      0.003251380519941449,
      0.0027281083166599274,
      0.002204835880547762,
      0.001681563793681562,
      0.001158291706815362,
      0.0006350195035338402,
      0.00011174729297636077,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0003849811910185963,
      0.0008885387214832008,
      0.001392096164636314,
      0.0018956535495817661,
      0.00239921105094254,
      0.002902768552303314,
      0.0034063260536640882,
      0.003132763085886836,
      0.0026481777895241976,
      0.0021635922603309155,
      0.0016790067311376333,
      0.0011944210855290294,
      0.0007098356145434082,
      0.00022525011445395648,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0
    ],
    [
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.0,
      0.000366741674952209,
      0.0008330700220540166,
      0.0012993983691558242,
      0.0017657268326729536,
      0.0022320549469441175,
      0.002698383294045925,
      0.0031647118739783764,
      0.003141313325613737,
      0.002692554146051407,
      0.0022437951993197203,
      0.00179503601975739,
      0.0013462770730257034,
      0.000897518009878695,
      0.0004487590049393475,
      0.0
    ]
  ],
  "n_fft": 400,
  "n_samples": 480000,
  "nb_max_frames": 3000,
  "padding_side": "right",
  "padding_value": 0.0,
  "processor_class": "WhisperProcessor",
  "return_attention_mask": false,
  "sampling_rate": 16000
}

[INFO|tokenization_utils_base.py:1797] 2022-12-20 10:22:02,861 >> loading file vocab.json
[INFO|tokenization_utils_base.py:1797] 2022-12-20 10:22:02,861 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:1797] 2022-12-20 10:22:02,861 >> loading file merges.txt
[INFO|tokenization_utils_base.py:1797] 2022-12-20 10:22:02,861 >> loading file normalizer.json
[INFO|tokenization_utils_base.py:1797] 2022-12-20 10:22:02,861 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:1797] 2022-12-20 10:22:02,861 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:1797] 2022-12-20 10:22:02,861 >> loading file tokenizer_config.json
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,917 >> Adding <|endoftext|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,917 >> Adding <|startoftranscript|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,917 >> Adding <|en|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,917 >> Adding <|zh|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,917 >> Adding <|de|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,917 >> Adding <|es|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,917 >> Adding <|ru|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|ko|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|fr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|ja|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|pt|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|tr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|pl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|ca|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|nl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|ar|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|sv|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|it|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|id|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|hi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|fi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|vi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|iw|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|uk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|el|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|ms|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,918 >> Adding <|cs|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|ro|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|da|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|hu|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|ta|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|no|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|th|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|ur|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|hr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|bg|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|lt|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|la|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|mi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|ml|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|cy|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|sk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|te|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|fa|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|lv|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|bn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|sr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|az|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,919 >> Adding <|sl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|kn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|et|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|mk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|br|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|eu|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|is|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|hy|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|ne|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|mn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|bs|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|kk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|sq|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|sw|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|gl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|mr|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|pa|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|si|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|km|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|sn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|yo|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|so|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,920 >> Adding <|af|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|oc|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|ka|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|be|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|tg|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|sd|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|gu|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|am|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|yi|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|lo|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|uz|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|fo|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|ht|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|ps|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|tk|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|nn|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|mt|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|sa|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|lb|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|my|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,921 >> Adding <|bo|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|tl|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|mg|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|as|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|tt|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|haw|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|ln|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|ha|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|ba|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|jw|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|su|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|translate|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|transcribe|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|startoflm|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|startofprev|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|nocaptions|> to the vocabulary
[INFO|tokenization_utils.py:426] 2022-12-20 10:22:02,922 >> Adding <|notimestamps|> to the vocabulary
/home/milan/whisper-audio-concat-test/./ is already a clone of https://huggingface.co/mikr/whisper-audio-concat-test. Make sure you pull the latest changes with `repo.git_pull()`.
12/20/2022 10:22:04 - WARNING - huggingface_hub.repository - /home/milan/whisper-audio-concat-test/./ is already a clone of https://huggingface.co/mikr/whisper-audio-concat-test. Make sure you pull the latest changes with `repo.git_pull()`.
[INFO|trainer.py:511] 2022-12-20 10:22:08,793 >> max_steps is given, it will override any value given in num_train_epochs
[INFO|trainer.py:565] 2022-12-20 10:22:08,794 >> Using cuda_amp half precision backend
[2022-12-20 10:22:08,801] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.7, git-hash=unknown, git-branch=unknown
[2022-12-20 10:22:09,703] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2022-12-20 10:22:10,934] [WARNING] [cpu_adam.py:83:__init__] FP16 params for CPUAdam may not work on AMD CPUs
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Using /home/milan/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/milan/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 3.2882912158966064 seconds
Adam Optimizer #0 is created with AVX2 arithmetic capability.
Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1
[2022-12-20 10:22:16,248] [INFO] [logging.py:68:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2022-12-20 10:22:16,551] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2022-12-20 10:22:16,551] [INFO] [utils.py:52:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2022-12-20 10:22:16,551] [INFO] [logging.py:68:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
[2022-12-20 10:22:16,551] [INFO] [stage_1_and_2.py:140:__init__] Reduce bucket size 200000000
[2022-12-20 10:22:16,552] [INFO] [stage_1_and_2.py:141:__init__] Allgather bucket size 200000000
[2022-12-20 10:22:16,552] [INFO] [stage_1_and_2.py:142:__init__] CPU Offload: True
[2022-12-20 10:22:16,552] [INFO] [stage_1_and_2.py:143:__init__] Round robin gradient partitioning: False
Using /home/milan/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Emitting ninja build file /home/milan/.cache/torch_extensions/py38_cu117/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module utils...
Time to load utils op: 0.6276412010192871 seconds
Rank: 0 partition count [1] and sizes[(1543304960, False)] 
[2022-12-20 10:22:20,332] [INFO] [utils.py:827:see_memory_usage] Before initializing optimizer states
[2022-12-20 10:22:20,333] [INFO] [utils.py:828:see_memory_usage] MA 3.0 GB         Max_MA 3.0 GB         CA 5.99 GB         Max_CA 6 GB 
[2022-12-20 10:22:20,333] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 15.43 GB, percent = 7.8%
[2022-12-20 10:22:24,263] [INFO] [utils.py:827:see_memory_usage] After initializing optimizer states
[2022-12-20 10:22:24,264] [INFO] [utils.py:828:see_memory_usage] MA 3.0 GB         Max_MA 3.0 GB         CA 5.99 GB         Max_CA 6 GB 
[2022-12-20 10:22:24,264] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 35.08 GB, percent = 17.8%
[2022-12-20 10:22:24,264] [INFO] [stage_1_and_2.py:525:__init__] optimizer state initialized
[2022-12-20 10:22:24,345] [INFO] [utils.py:827:see_memory_usage] After initializing ZeRO optimizer
[2022-12-20 10:22:24,346] [INFO] [utils.py:828:see_memory_usage] MA 3.0 GB         Max_MA 3.0 GB         CA 5.99 GB         Max_CA 6 GB 
[2022-12-20 10:22:24,346] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 35.08 GB, percent = 17.8%
[2022-12-20 10:22:24,379] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2022-12-20 10:22:24,380] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupDecayLR
[2022-12-20 10:22:24,380] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupDecayLR object at 0x7ff9c20fbfd0>
[2022-12-20 10:22:24,380] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05], mom=[[0.9, 0.999]]
[2022-12-20 10:22:24,382] [INFO] [config.py:1020:print] DeepSpeedEngine configuration:
[2022-12-20 10:22:24,382] [INFO] [config.py:1024:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2022-12-20 10:22:24,382] [INFO] [config.py:1024:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2022-12-20 10:22:24,382] [INFO] [config.py:1024:print]   amp_enabled .................. False
[2022-12-20 10:22:24,382] [INFO] [config.py:1024:print]   amp_params ................... False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   bfloat16_enabled ............. False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   checkpoint_parallel_write_pipeline  False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   checkpoint_tag_validation_enabled  True
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   checkpoint_tag_validation_fail  False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7ff94f5c16a0>
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   communication_data_type ...... None
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   curriculum_enabled ........... False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   curriculum_params ............ False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   dataloader_drop_last ......... False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   disable_allgather ............ False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   dump_state ................... False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   dynamic_loss_scale_args ...... {'init_scale': 65536, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   eigenvalue_enabled ........... False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   eigenvalue_gas_boundary_resolution  1
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   eigenvalue_layer_num ......... 0
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   eigenvalue_max_iter .......... 100
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   eigenvalue_stability ......... 1e-06
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   eigenvalue_tol ............... 0.01
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   eigenvalue_verbose ........... False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   elasticity_enabled ........... False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   flops_profiler_config ........ {
    "enabled": false, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   fp16_auto_cast ............... False
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   fp16_enabled ................. True
[2022-12-20 10:22:24,383] [INFO] [config.py:1024:print]   fp16_master_weights_and_gradients  False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   global_rank .................. 0
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   grad_accum_dtype ............. None
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   gradient_accumulation_steps .. 2
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   gradient_clipping ............ 1.0
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   gradient_predivide_factor .... 1.0
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   initial_dynamic_scale ........ 65536
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   load_universal_checkpoint .... False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   loss_scale ................... 0
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   memory_breakdown ............. False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   monitor_config ............... <deepspeed.monitor.config.DeepSpeedMonitorConfig object at 0x7ff94f5c1910>
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   optimizer_legacy_fusion ...... False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   optimizer_name ............... adamw
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   optimizer_params ............. {'lr': 1e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   pld_enabled .................. False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   pld_params ................... False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   prescale_gradients ........... False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   scheduler_name ............... WarmupDecayLR
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   scheduler_params ............. {'last_batch_iteration': -1, 'total_num_steps': 5000, 'warmup_min_lr': 0, 'warmup_max_lr': 1e-05, 'warmup_num_steps': 500}
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   sparse_attention ............. None
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   sparse_gradients_enabled ..... False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   steps_per_print .............. 10
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   train_batch_size ............. 64
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   train_micro_batch_size_per_gpu  32
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   use_node_local_storage ....... False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   wall_clock_breakdown ......... False
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   world_size ................... 1
[2022-12-20 10:22:24,384] [INFO] [config.py:1024:print]   zero_allow_untested_optimizer  False
[2022-12-20 10:22:24,385] [INFO] [config.py:1024:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False
[2022-12-20 10:22:24,385] [INFO] [config.py:1024:print]   zero_enabled ................. True
[2022-12-20 10:22:24,385] [INFO] [config.py:1024:print]   zero_optimization_stage ...... 2
[2022-12-20 10:22:24,385] [INFO] [config.py:1009:print_user_config]   json = {
    "fp16": {
        "enabled": true, 
        "loss_scale": 0, 
        "loss_scale_window": 1000, 
        "initial_scale_power": 16, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 1e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.0
        }
    }, 
    "scheduler": {
        "type": "WarmupDecayLR", 
        "params": {
            "last_batch_iteration": -1, 
            "total_num_steps": 5.000000e+03, 
            "warmup_min_lr": 0, 
            "warmup_max_lr": 1e-05, 
            "warmup_num_steps": 500
        }
    }, 
    "zero_optimization": {
        "stage": 2, 
        "offload_optimizer": {
            "device": "cpu", 
            "pin_memory": true
        }, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 2.000000e+08, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 2.000000e+08, 
        "contiguous_gradients": true
    }, 
    "gradient_accumulation_steps": 2, 
    "gradient_clipping": 1.0, 
    "train_batch_size": 64, 
    "train_micro_batch_size_per_gpu": 32
}
Using /home/milan/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00040650367736816406 seconds
[INFO|trainer.py:1641] 2022-12-20 10:22:24,388 >> ***** Running training *****
[INFO|trainer.py:1642] 2022-12-20 10:22:24,388 >>   Num examples = 2650
[INFO|trainer.py:1643] 2022-12-20 10:22:24,388 >>   Num Epochs = 122
[INFO|trainer.py:1644] 2022-12-20 10:22:24,388 >>   Instantaneous batch size per device = 32
[INFO|trainer.py:1645] 2022-12-20 10:22:24,388 >>   Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:1646] 2022-12-20 10:22:24,388 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1647] 2022-12-20 10:22:24,388 >>   Total optimization steps = 5000
[INFO|trainer.py:1648] 2022-12-20 10:22:24,392 >>   Number of trainable parameters = 1543304960

  0%|          | 0/5000 [00:00<?, ?it/s]
  0%|          | 1/5000 [00:13<18:25:54, 13.27s/it][2022-12-20 10:22:48,893] [INFO] [timer.py:197:stop] 0/4, RunningAvgSamplesPerSec=5.8814545641618095, CurrSamplesPerSec=5.346103229416477, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 2/5000 [00:24<16:44:48, 12.06s/it][2022-12-20 10:22:59,896] [INFO] [timer.py:197:stop] 0/6, RunningAvgSamplesPerSec=5.883145480351049, CurrSamplesPerSec=5.345442120988815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 3/5000 [00:35<16:04:18, 11.58s/it][2022-12-20 10:23:11,271] [INFO] [timer.py:197:stop] 0/8, RunningAvgSamplesPerSec=5.8176665539142975, CurrSamplesPerSec=5.012736083343545, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 4/5000 [00:46<15:57:25, 11.50s/it][2022-12-20 10:23:21,706] [INFO] [stage_1_and_2.py:1765:step] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, reducing to 65536
[2022-12-20 10:23:21,707] [INFO] [timer.py:197:stop] 0/10, RunningAvgSamplesPerSec=5.910662394854158, CurrSamplesPerSec=5.923662200423984, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 5/5000 [00:57<15:25:20, 11.12s/it][2022-12-20 10:23:32,717] [INFO] [timer.py:197:stop] 0/12, RunningAvgSamplesPerSec=5.904443044106101, CurrSamplesPerSec=5.348837546653676, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 6/5000 [01:08<15:22:10, 11.08s/it][2022-12-20 10:23:43,755] [INFO] [timer.py:197:stop] 0/14, RunningAvgSamplesPerSec=5.897781352467236, CurrSamplesPerSec=5.314892432588881, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 7/5000 [01:19<15:20:52, 11.07s/it][2022-12-20 10:23:54,728] [INFO] [timer.py:197:stop] 0/16, RunningAvgSamplesPerSec=5.8981574973826065, CurrSamplesPerSec=5.338246310663274, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 8/5000 [01:30<15:18:14, 11.04s/it][2022-12-20 10:24:05,789] [INFO] [timer.py:197:stop] 0/18, RunningAvgSamplesPerSec=5.892644728106006, CurrSamplesPerSec=5.313068961651518, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 9/5000 [01:41<15:18:40, 11.04s/it][2022-12-20 10:24:16,792] [INFO] [logging.py:68:log_dist] [Rank 0] step=10, skipped=1, lr=[3.535580269163017e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:24:16,794] [INFO] [timer.py:197:stop] 0/20, RunningAvgSamplesPerSec=5.8916641634587705, CurrSamplesPerSec=5.345411890632343, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 10/5000 [01:52<15:17:29, 11.03s/it][2022-12-20 10:24:27,819] [INFO] [timer.py:197:stop] 0/22, RunningAvgSamplesPerSec=5.889766748036226, CurrSamplesPerSec=5.309852379886293, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 11/5000 [02:03<15:17:08, 11.03s/it][2022-12-20 10:24:38,829] [INFO] [timer.py:197:stop] 0/24, RunningAvgSamplesPerSec=5.888990416214062, CurrSamplesPerSec=5.307394070082909, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 12/5000 [02:14<15:16:26, 11.02s/it][2022-12-20 10:24:49,913] [INFO] [timer.py:197:stop] 0/26, RunningAvgSamplesPerSec=5.884991434584374, CurrSamplesPerSec=5.306818036243231, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 13/5000 [02:25<15:17:46, 11.04s/it][2022-12-20 10:25:01,101] [INFO] [timer.py:197:stop] 0/28, RunningAvgSamplesPerSec=5.877292121381068, CurrSamplesPerSec=5.130267423634092, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 14/5000 [02:36<15:21:16, 11.09s/it][2022-12-20 10:25:12,119] [INFO] [timer.py:197:stop] 0/30, RunningAvgSamplesPerSec=5.877284079532008, CurrSamplesPerSec=5.332734069349443, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 15/5000 [02:47<15:19:21, 11.07s/it][2022-12-20 10:25:23,167] [INFO] [timer.py:197:stop] 0/32, RunningAvgSamplesPerSec=5.876699417402653, CurrSamplesPerSec=5.318107328766784, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 16/5000 [02:58<15:18:45, 11.06s/it][2022-12-20 10:25:34,380] [INFO] [timer.py:197:stop] 0/34, RunningAvgSamplesPerSec=5.870134737880074, CurrSamplesPerSec=5.3123356762705525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 17/5000 [03:09<15:22:22, 11.11s/it][2022-12-20 10:25:45,438] [INFO] [timer.py:197:stop] 0/36, RunningAvgSamplesPerSec=5.869268480921617, CurrSamplesPerSec=5.297387426539539, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 18/5000 [03:21<15:20:59, 11.09s/it][2022-12-20 10:25:56,452] [INFO] [timer.py:197:stop] 0/38, RunningAvgSamplesPerSec=5.869781989968984, CurrSamplesPerSec=5.301425547521082, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 19/5000 [03:32<15:18:53, 11.07s/it][2022-12-20 10:26:07,486] [INFO] [logging.py:68:log_dist] [Rank 0] step=20, skipped=1, lr=[4.737931873634953e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:26:07,488] [INFO] [timer.py:197:stop] 0/40, RunningAvgSamplesPerSec=5.870117333953952, CurrSamplesPerSec=5.337023637625309, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 20/5000 [03:43<15:17:51, 11.06s/it][2022-12-20 10:26:18,557] [INFO] [timer.py:197:stop] 0/42, RunningAvgSamplesPerSec=5.86906144518976, CurrSamplesPerSec=5.299037167362037, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 21/5000 [03:54<15:17:55, 11.06s/it][2022-12-20 10:26:29,715] [INFO] [timer.py:197:stop] 0/44, RunningAvgSamplesPerSec=5.8658397725575995, CurrSamplesPerSec=5.2302204149300495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 22/5000 [04:05<15:20:09, 11.09s/it][2022-12-20 10:26:40,727] [INFO] [timer.py:197:stop] 0/46, RunningAvgSamplesPerSec=5.866507269104006, CurrSamplesPerSec=5.308386736481321, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 23/5000 [04:16<15:18:00, 11.07s/it][2022-12-20 10:26:51,771] [INFO] [timer.py:197:stop] 0/48, RunningAvgSamplesPerSec=5.866358292982727, CurrSamplesPerSec=5.296070546653146, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 24/5000 [04:27<15:17:14, 11.06s/it][2022-12-20 10:27:02,870] [INFO] [timer.py:197:stop] 0/50, RunningAvgSamplesPerSec=5.86497698207735, CurrSamplesPerSec=5.331737995660014, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  0%|          | 25/5000 [04:38<15:18:02, 11.07s/it]
                                                    
{'loss': 0.2509, 'learning_rate': 5.113844316514203e-06, 'epoch': 0.6}

  0%|          | 25/5000 [04:38<15:18:02, 11.07s/it][2022-12-20 10:27:13,911] [INFO] [timer.py:197:stop] 0/52, RunningAvgSamplesPerSec=5.864995428656389, CurrSamplesPerSec=5.306736205514212, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 26/5000 [04:49<15:17:04, 11.06s/it][2022-12-20 10:27:25,025] [INFO] [timer.py:197:stop] 0/54, RunningAvgSamplesPerSec=5.8634706103271, CurrSamplesPerSec=5.280355882025363, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 27/5000 [05:00<15:18:10, 11.08s/it][2022-12-20 10:27:36,058] [INFO] [timer.py:197:stop] 0/56, RunningAvgSamplesPerSec=5.8636823916423575, CurrSamplesPerSec=5.324947001558154, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 28/5000 [05:11<15:16:51, 11.06s/it][2022-12-20 10:27:47,117] [INFO] [timer.py:197:stop] 0/58, RunningAvgSamplesPerSec=5.863351765084911, CurrSamplesPerSec=5.331683563395017, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 29/5000 [05:22<15:16:33, 11.06s/it][2022-12-20 10:27:58,150] [INFO] [logging.py:68:log_dist] [Rank 0] step=30, skipped=1, lr=[5.418355874832054e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:27:58,152] [INFO] [timer.py:197:stop] 0/60, RunningAvgSamplesPerSec=5.863501012868562, CurrSamplesPerSec=5.333975339384993, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 30/5000 [05:33<15:15:40, 11.05s/it][2022-12-20 10:28:09,222] [INFO] [timer.py:197:stop] 0/62, RunningAvgSamplesPerSec=5.8630126666776965, CurrSamplesPerSec=5.310446091748985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 31/5000 [05:44<15:15:52, 11.06s/it][2022-12-20 10:28:19,685] [INFO] [stage_1_and_2.py:1765:step] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, reducing to 32768.0
[2022-12-20 10:28:19,688] [INFO] [timer.py:197:stop] 0/64, RunningAvgSamplesPerSec=5.873030860997967, CurrSamplesPerSec=5.886545979759184, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 32/5000 [05:55<15:00:57, 10.88s/it][2022-12-20 10:28:30,768] [INFO] [timer.py:197:stop] 0/66, RunningAvgSamplesPerSec=5.872174951085589, CurrSamplesPerSec=5.298567739971801, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 33/5000 [06:06<15:05:43, 10.94s/it][2022-12-20 10:28:41,796] [INFO] [timer.py:197:stop] 0/68, RunningAvgSamplesPerSec=5.872359164459001, CurrSamplesPerSec=5.315037025708472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 34/5000 [06:17<15:07:43, 10.97s/it][2022-12-20 10:28:52,895] [INFO] [timer.py:197:stop] 0/70, RunningAvgSamplesPerSec=5.871356619893246, CurrSamplesPerSec=5.279161864784775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 35/5000 [06:28<15:10:47, 11.01s/it][2022-12-20 10:29:03,978] [INFO] [timer.py:197:stop] 0/72, RunningAvgSamplesPerSec=5.870487356198055, CurrSamplesPerSec=5.295958537569409, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 36/5000 [06:39<15:12:31, 11.03s/it][2022-12-20 10:29:15,036] [INFO] [timer.py:197:stop] 0/74, RunningAvgSamplesPerSec=5.870289500098377, CurrSamplesPerSec=5.305059860602074, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 37/5000 [06:50<15:13:01, 11.04s/it][2022-12-20 10:29:26,151] [INFO] [timer.py:197:stop] 0/76, RunningAvgSamplesPerSec=5.86929775315288, CurrSamplesPerSec=5.313538437858181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 38/5000 [07:01<15:14:44, 11.06s/it][2022-12-20 10:29:37,244] [INFO] [timer.py:197:stop] 0/78, RunningAvgSamplesPerSec=5.868419656882571, CurrSamplesPerSec=5.294760169774455, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 39/5000 [07:12<15:15:21, 11.07s/it][2022-12-20 10:29:48,359] [INFO] [logging.py:68:log_dist] [Rank 0] step=40, skipped=2, lr=[5.853283267612517e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:29:48,360] [INFO] [timer.py:197:stop] 0/80, RunningAvgSamplesPerSec=5.867269334314438, CurrSamplesPerSec=5.2257062064624185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 40/5000 [07:23<15:16:18, 11.08s/it][2022-12-20 10:29:59,465] [INFO] [timer.py:197:stop] 0/82, RunningAvgSamplesPerSec=5.8665526220186335, CurrSamplesPerSec=5.285316757954975, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 41/5000 [07:35<15:16:37, 11.09s/it][2022-12-20 10:30:09,642] [INFO] [timer.py:197:stop] 0/84, RunningAvgSamplesPerSec=5.877687597966856, CurrSamplesPerSec=5.280113254878277, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 42/5000 [07:50<16:57:08, 12.31s/it][2022-12-20 10:30:20,863] [INFO] [timer.py:197:stop] 0/86, RunningAvgSamplesPerSec=5.874998502236885, CurrSamplesPerSec=5.176902001769325, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 43/5000 [08:01<16:29:30, 11.98s/it][2022-12-20 10:30:31,925] [INFO] [timer.py:197:stop] 0/88, RunningAvgSamplesPerSec=5.874420158809555, CurrSamplesPerSec=5.296744581932894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 44/5000 [08:12<16:07:23, 11.71s/it][2022-12-20 10:30:43,008] [INFO] [timer.py:197:stop] 0/90, RunningAvgSamplesPerSec=5.8738377277486, CurrSamplesPerSec=5.305843784965715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 45/5000 [08:23<15:57:37, 11.60s/it][2022-12-20 10:30:54,278] [INFO] [timer.py:197:stop] 0/92, RunningAvgSamplesPerSec=5.870826853395746, CurrSamplesPerSec=5.354598254312376, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 46/5000 [08:34<15:41:45, 11.41s/it][2022-12-20 10:31:05,311] [INFO] [timer.py:197:stop] 0/94, RunningAvgSamplesPerSec=5.870810611016451, CurrSamplesPerSec=5.298239148452526, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 47/5000 [08:45<15:33:16, 11.31s/it][2022-12-20 10:31:16,605] [INFO] [timer.py:197:stop] 0/96, RunningAvgSamplesPerSec=5.867718940086736, CurrSamplesPerSec=5.104541928824625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 48/5000 [08:57<15:33:03, 11.31s/it][2022-12-20 10:31:27,632] [INFO] [timer.py:197:stop] 0/98, RunningAvgSamplesPerSec=5.8677471073657035, CurrSamplesPerSec=5.340984284219985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 49/5000 [09:08<15:26:10, 11.22s/it][2022-12-20 10:31:38,677] [INFO] [logging.py:68:log_dist] [Rank 0] step=50, skipped=2, lr=[6.229195710491767e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:31:38,679] [INFO] [timer.py:197:stop] 0/100, RunningAvgSamplesPerSec=5.867724434871806, CurrSamplesPerSec=5.329774266221371, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 50/5000 [09:19<15:21:07, 11.17s/it]
                                                    
{'loss': 0.1429, 'learning_rate': 6.229195710491767e-06, 'epoch': 1.22}

  1%|          | 50/5000 [09:19<15:21:07, 11.17s/it][2022-12-20 10:31:49,761] [INFO] [timer.py:197:stop] 0/102, RunningAvgSamplesPerSec=5.867238317470352, CurrSamplesPerSec=5.288021734602139, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 51/5000 [09:30<15:19:39, 11.15s/it][2022-12-20 10:32:00,834] [INFO] [timer.py:197:stop] 0/104, RunningAvgSamplesPerSec=5.866787226210247, CurrSamplesPerSec=5.317731432108492, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 52/5000 [09:41<15:16:22, 11.11s/it][2022-12-20 10:32:12,018] [INFO] [timer.py:197:stop] 0/106, RunningAvgSamplesPerSec=5.865216995983049, CurrSamplesPerSec=5.180955039650688, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 53/5000 [09:52<15:19:04, 11.15s/it][2022-12-20 10:32:23,107] [INFO] [timer.py:197:stop] 0/108, RunningAvgSamplesPerSec=5.864828483742427, CurrSamplesPerSec=5.300241867612233, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 54/5000 [10:03<15:15:47, 11.11s/it][2022-12-20 10:32:34,109] [INFO] [timer.py:197:stop] 0/110, RunningAvgSamplesPerSec=5.865157642277969, CurrSamplesPerSec=5.317506003887606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 55/5000 [10:14<15:13:01, 11.08s/it][2022-12-20 10:32:45,161] [INFO] [timer.py:197:stop] 0/112, RunningAvgSamplesPerSec=5.864996990605605, CurrSamplesPerSec=5.276864035023055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 56/5000 [10:25<15:13:25, 11.09s/it][2022-12-20 10:32:56,219] [INFO] [timer.py:197:stop] 0/114, RunningAvgSamplesPerSec=5.8647831952686795, CurrSamplesPerSec=5.315040603803902, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 57/5000 [10:36<15:11:19, 11.06s/it][2022-12-20 10:33:07,393] [INFO] [timer.py:197:stop] 0/116, RunningAvgSamplesPerSec=5.863486544752744, CurrSamplesPerSec=5.173499531574897, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 58/5000 [10:47<15:14:57, 11.11s/it][2022-12-20 10:33:18,442] [INFO] [timer.py:197:stop] 0/118, RunningAvgSamplesPerSec=5.863615945144509, CurrSamplesPerSec=5.316347983403619, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 59/5000 [10:59<15:13:48, 11.10s/it][2022-12-20 10:33:29,547] [INFO] [logging.py:68:log_dist] [Rank 0] step=60, skipped=2, lr=[6.533707268809618e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:33:29,549] [INFO] [timer.py:197:stop] 0/120, RunningAvgSamplesPerSec=5.8631475955142855, CurrSamplesPerSec=5.297602788239249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 60/5000 [11:10<15:13:32, 11.10s/it][2022-12-20 10:33:40,700] [INFO] [timer.py:197:stop] 0/122, RunningAvgSamplesPerSec=5.862134257826161, CurrSamplesPerSec=5.232823371989121, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 61/5000 [11:21<15:14:24, 11.11s/it][2022-12-20 10:33:51,727] [INFO] [timer.py:197:stop] 0/124, RunningAvgSamplesPerSec=5.86241089864623, CurrSamplesPerSec=5.3309576237300975, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|          | 62/5000 [11:32<15:11:43, 11.08s/it][2022-12-20 10:34:02,753] [INFO] [timer.py:197:stop] 0/126, RunningAvgSamplesPerSec=5.862531509458954, CurrSamplesPerSec=5.313768999559752, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 63/5000 [11:43<15:11:43, 11.08s/it][2022-12-20 10:34:13,879] [INFO] [timer.py:197:stop] 0/128, RunningAvgSamplesPerSec=5.8617915652725525, CurrSamplesPerSec=5.278223686821037, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 64/5000 [11:54<15:11:17, 11.08s/it][2022-12-20 10:34:24,906] [INFO] [timer.py:197:stop] 0/130, RunningAvgSamplesPerSec=5.861911770276758, CurrSamplesPerSec=5.316452433086607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 65/5000 [12:05<15:09:23, 11.06s/it][2022-12-20 10:34:35,928] [INFO] [timer.py:197:stop] 0/132, RunningAvgSamplesPerSec=5.862066764465409, CurrSamplesPerSec=5.30377018219087, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 66/5000 [12:16<15:09:59, 11.07s/it][2022-12-20 10:34:46,983] [INFO] [timer.py:197:stop] 0/134, RunningAvgSamplesPerSec=5.861953050205013, CurrSamplesPerSec=5.333423192160665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 67/5000 [12:27<15:08:17, 11.05s/it][2022-12-20 10:34:58,037] [INFO] [timer.py:197:stop] 0/136, RunningAvgSamplesPerSec=5.862085406687506, CurrSamplesPerSec=5.303489353604265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 68/5000 [12:38<15:09:11, 11.06s/it][2022-12-20 10:35:09,403] [INFO] [timer.py:197:stop] 0/138, RunningAvgSamplesPerSec=5.859516821971333, CurrSamplesPerSec=5.059717056677104, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 69/5000 [12:49<15:16:06, 11.15s/it][2022-12-20 10:35:20,404] [INFO] [logging.py:68:log_dist] [Rank 0] step=70, skipped=2, lr=[6.7896601657751925e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:35:20,405] [INFO] [timer.py:197:stop] 0/140, RunningAvgSamplesPerSec=5.859859108396585, CurrSamplesPerSec=5.353309148506309, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 70/5000 [13:00<15:12:47, 11.11s/it][2022-12-20 10:35:31,504] [INFO] [timer.py:197:stop] 0/142, RunningAvgSamplesPerSec=5.85956265448322, CurrSamplesPerSec=5.283152896686594, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 71/5000 [13:12<15:19:28, 11.19s/it][2022-12-20 10:35:42,842] [INFO] [timer.py:197:stop] 0/144, RunningAvgSamplesPerSec=5.857360277259472, CurrSamplesPerSec=5.328067050781999, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 72/5000 [13:23<15:15:41, 11.15s/it][2022-12-20 10:35:53,855] [INFO] [timer.py:197:stop] 0/146, RunningAvgSamplesPerSec=5.8577641546740615, CurrSamplesPerSec=5.373210580896251, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 73/5000 [13:34<15:11:16, 11.10s/it][2022-12-20 10:36:05,258] [INFO] [timer.py:197:stop] 0/148, RunningAvgSamplesPerSec=5.855164474841027, CurrSamplesPerSec=5.0012772812146356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  1%|▏         | 74/5000 [13:45<15:18:09, 11.18s/it][2022-12-20 10:36:16,282] [INFO] [timer.py:197:stop] 0/150, RunningAvgSamplesPerSec=5.855386012044155, CurrSamplesPerSec=5.300191634669637, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 75/5000 [13:56<15:15:14, 11.15s/it]
                                                    
{'loss': 0.0992, 'learning_rate': 6.903829450223392e-06, 'epoch': 1.82}

  2%|▏         | 75/5000 [13:56<15:15:14, 11.15s/it][2022-12-20 10:36:27,306] [INFO] [timer.py:197:stop] 0/152, RunningAvgSamplesPerSec=5.855668675712257, CurrSamplesPerSec=5.3431396321531235, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 76/5000 [14:07<15:12:24, 11.12s/it][2022-12-20 10:36:38,414] [INFO] [timer.py:197:stop] 0/154, RunningAvgSamplesPerSec=5.855402477126268, CurrSamplesPerSec=5.298864783291435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 77/5000 [14:18<15:11:14, 11.11s/it][2022-12-20 10:36:49,480] [INFO] [timer.py:197:stop] 0/156, RunningAvgSamplesPerSec=5.8553214508026405, CurrSamplesPerSec=5.295619404626892, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 78/5000 [14:30<15:09:33, 11.09s/it][2022-12-20 10:37:00,505] [INFO] [timer.py:197:stop] 0/158, RunningAvgSamplesPerSec=5.8555155701103985, CurrSamplesPerSec=5.312547839270027, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 79/5000 [14:41<15:08:32, 11.08s/it][2022-12-20 10:37:11,541] [INFO] [logging.py:68:log_dist] [Rank 0] step=80, skipped=2, lr=[7.010432126517687e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:37:11,543] [INFO] [timer.py:197:stop] 0/160, RunningAvgSamplesPerSec=5.855637308805398, CurrSamplesPerSec=5.326441886114601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 80/5000 [14:52<15:06:58, 11.06s/it][2022-12-20 10:37:22,548] [INFO] [timer.py:197:stop] 0/162, RunningAvgSamplesPerSec=5.85595548292032, CurrSamplesPerSec=5.340737753718186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 81/5000 [15:03<15:06:37, 11.06s/it][2022-12-20 10:37:33,727] [INFO] [timer.py:197:stop] 0/164, RunningAvgSamplesPerSec=5.8553479497823036, CurrSamplesPerSec=5.259307529244275, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 82/5000 [15:14<15:08:00, 11.08s/it][2022-12-20 10:37:43,831] [INFO] [timer.py:197:stop] 0/166, RunningAvgSamplesPerSec=5.861494518397765, CurrSamplesPerSec=6.265858499469083, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 10:37:54,868] [INFO] [timer.py:197:stop] 0/168, RunningAvgSamplesPerSec=5.861572248825517, CurrSamplesPerSec=5.294247644578068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 83/5000 [15:30<17:13:43, 12.61s/it][2022-12-20 10:38:06,000] [INFO] [timer.py:197:stop] 0/170, RunningAvgSamplesPerSec=5.861134287633014, CurrSamplesPerSec=5.264067460272326, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 84/5000 [15:41<16:37:04, 12.17s/it][2022-12-20 10:38:17,033] [INFO] [timer.py:197:stop] 0/172, RunningAvgSamplesPerSec=5.861187628179735, CurrSamplesPerSec=5.307569108412553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 85/5000 [15:52<16:08:57, 11.83s/it][2022-12-20 10:38:28,089] [INFO] [timer.py:197:stop] 0/174, RunningAvgSamplesPerSec=5.861102495635817, CurrSamplesPerSec=5.306581153695259, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 86/5000 [16:03<15:49:46, 11.60s/it][2022-12-20 10:38:39,213] [INFO] [timer.py:197:stop] 0/176, RunningAvgSamplesPerSec=5.860798879131059, CurrSamplesPerSec=5.298156536372784, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 87/5000 [16:14<15:37:57, 11.45s/it][2022-12-20 10:38:50,230] [INFO] [timer.py:197:stop] 0/178, RunningAvgSamplesPerSec=5.860957146837867, CurrSamplesPerSec=5.306349746791664, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 88/5000 [16:25<15:27:01, 11.32s/it][2022-12-20 10:39:01,375] [INFO] [logging.py:68:log_dist] [Rank 0] step=90, skipped=2, lr=[7.204536060149867e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:39:01,376] [INFO] [timer.py:197:stop] 0/180, RunningAvgSamplesPerSec=5.860328646378536, CurrSamplesPerSec=5.230492518324473, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 89/5000 [16:36<15:22:28, 11.27s/it][2022-12-20 10:39:12,399] [INFO] [timer.py:197:stop] 0/182, RunningAvgSamplesPerSec=5.860456149765244, CurrSamplesPerSec=5.353855383623674, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 90/5000 [16:47<15:16:13, 11.20s/it][2022-12-20 10:39:23,472] [INFO] [timer.py:197:stop] 0/184, RunningAvgSamplesPerSec=5.860386866049039, CurrSamplesPerSec=5.319645813872818, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 91/5000 [16:59<15:13:00, 11.16s/it][2022-12-20 10:39:34,578] [INFO] [timer.py:197:stop] 0/186, RunningAvgSamplesPerSec=5.860032107959842, CurrSamplesPerSec=5.310408061638177, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 92/5000 [17:10<15:11:29, 11.14s/it][2022-12-20 10:39:45,682] [INFO] [timer.py:197:stop] 0/188, RunningAvgSamplesPerSec=5.85969361134499, CurrSamplesPerSec=5.27803687995338, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 93/5000 [17:21<15:10:21, 11.13s/it][2022-12-20 10:39:56,838] [INFO] [timer.py:197:stop] 0/190, RunningAvgSamplesPerSec=5.859055678801858, CurrSamplesPerSec=5.221323886708502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 94/5000 [17:32<15:10:47, 11.14s/it][2022-12-20 10:40:07,904] [INFO] [timer.py:197:stop] 0/192, RunningAvgSamplesPerSec=5.859008110900134, CurrSamplesPerSec=5.323973263380329, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 95/5000 [17:43<15:08:49, 11.12s/it][2022-12-20 10:40:18,924] [INFO] [timer.py:197:stop] 0/194, RunningAvgSamplesPerSec=5.859157663391512, CurrSamplesPerSec=5.312659289457989, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 96/5000 [17:54<15:06:14, 11.09s/it][2022-12-20 10:40:30,020] [INFO] [timer.py:197:stop] 0/196, RunningAvgSamplesPerSec=5.858885625976701, CurrSamplesPerSec=5.257421699832881, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 97/5000 [18:05<15:06:15, 11.09s/it][2022-12-20 10:40:41,080] [INFO] [timer.py:197:stop] 0/198, RunningAvgSamplesPerSec=5.858806748355723, CurrSamplesPerSec=5.302112676028519, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 98/5000 [18:16<15:05:20, 11.08s/it][2022-12-20 10:40:52,106] [INFO] [logging.py:68:log_dist] [Rank 0] step=100, skipped=2, lr=[7.377725845391017e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:40:52,108] [INFO] [timer.py:197:stop] 0/200, RunningAvgSamplesPerSec=5.858910664276977, CurrSamplesPerSec=5.3371129841647065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 99/5000 [18:27<15:03:50, 11.07s/it][2022-12-20 10:41:03,144] [INFO] [timer.py:197:stop] 0/202, RunningAvgSamplesPerSec=5.858967157618062, CurrSamplesPerSec=5.321975786225992, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 100/5000 [18:38<15:02:56, 11.06s/it]
                                                     
{'loss': 0.0708, 'learning_rate': 7.394062147380189e-06, 'epoch': 2.43}

  2%|▏         | 100/5000 [18:38<15:02:56, 11.06s/it][2022-12-20 10:41:14,233] [INFO] [timer.py:197:stop] 0/204, RunningAvgSamplesPerSec=5.858839422992998, CurrSamplesPerSec=5.290660867627884, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 101/5000 [18:49<15:03:33, 11.07s/it][2022-12-20 10:41:25,398] [INFO] [timer.py:197:stop] 0/206, RunningAvgSamplesPerSec=5.858216268012085, CurrSamplesPerSec=5.175106122895775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 102/5000 [19:00<15:05:47, 11.10s/it][2022-12-20 10:41:36,466] [INFO] [timer.py:197:stop] 0/208, RunningAvgSamplesPerSec=5.858106742467774, CurrSamplesPerSec=5.307740590009921, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 103/5000 [19:12<15:04:55, 11.09s/it][2022-12-20 10:41:47,534] [INFO] [timer.py:197:stop] 0/210, RunningAvgSamplesPerSec=5.8580017499320585, CurrSamplesPerSec=5.303705211739256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 104/5000 [19:23<15:04:16, 11.08s/it][2022-12-20 10:41:58,536] [INFO] [timer.py:197:stop] 0/212, RunningAvgSamplesPerSec=5.858326882634229, CurrSamplesPerSec=5.362766316878288, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 105/5000 [19:34<15:02:07, 11.06s/it][2022-12-20 10:42:09,586] [INFO] [timer.py:197:stop] 0/214, RunningAvgSamplesPerSec=5.858310306464573, CurrSamplesPerSec=5.316025815679389, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 106/5000 [19:45<15:01:45, 11.06s/it][2022-12-20 10:42:20,708] [INFO] [timer.py:197:stop] 0/216, RunningAvgSamplesPerSec=5.857937908596017, CurrSamplesPerSec=5.211202201057734, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 107/5000 [19:56<15:03:18, 11.08s/it][2022-12-20 10:42:31,835] [INFO] [timer.py:197:stop] 0/218, RunningAvgSamplesPerSec=5.857920788827646, CurrSamplesPerSec=5.317615766082435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 108/5000 [20:07<15:04:15, 11.09s/it][2022-12-20 10:42:42,883] [INFO] [logging.py:68:log_dist] [Rank 0] step=110, skipped=2, lr=[7.5340731916996546e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:42:42,884] [INFO] [timer.py:197:stop] 0/220, RunningAvgSamplesPerSec=5.8580119599976825, CurrSamplesPerSec=5.285361297891226, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 109/5000 [20:18<15:03:02, 11.08s/it][2022-12-20 10:42:54,235] [INFO] [timer.py:197:stop] 0/222, RunningAvgSamplesPerSec=5.85654112936648, CurrSamplesPerSec=5.060507989795864, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 110/5000 [20:29<15:09:31, 11.16s/it][2022-12-20 10:43:05,260] [INFO] [timer.py:197:stop] 0/224, RunningAvgSamplesPerSec=5.856660090001322, CurrSamplesPerSec=5.30521356533897, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 111/5000 [20:40<15:06:04, 11.12s/it][2022-12-20 10:43:16,300] [INFO] [timer.py:197:stop] 0/226, RunningAvgSamplesPerSec=5.856765560544798, CurrSamplesPerSec=5.31082453153398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 112/5000 [20:51<15:03:55, 11.10s/it][2022-12-20 10:43:27,445] [INFO] [timer.py:197:stop] 0/228, RunningAvgSamplesPerSec=5.856322559853867, CurrSamplesPerSec=5.296503790564597, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 113/5000 [21:03<15:04:55, 11.11s/it][2022-12-20 10:43:38,502] [INFO] [timer.py:197:stop] 0/230, RunningAvgSamplesPerSec=5.856296525375066, CurrSamplesPerSec=5.281093040293156, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 114/5000 [21:14<15:03:27, 11.09s/it][2022-12-20 10:43:49,553] [INFO] [timer.py:197:stop] 0/232, RunningAvgSamplesPerSec=5.856304595546012, CurrSamplesPerSec=5.307485575491324, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 115/5000 [21:25<15:02:11, 11.08s/it][2022-12-20 10:44:00,588] [INFO] [timer.py:197:stop] 0/234, RunningAvgSamplesPerSec=5.856382518994549, CurrSamplesPerSec=5.317182431925186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 116/5000 [21:36<15:00:53, 11.07s/it][2022-12-20 10:44:11,665] [INFO] [timer.py:197:stop] 0/236, RunningAvgSamplesPerSec=5.85626210034176, CurrSamplesPerSec=5.2831329327552305, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 117/5000 [21:47<15:00:56, 11.07s/it][2022-12-20 10:44:22,768] [INFO] [timer.py:197:stop] 0/238, RunningAvgSamplesPerSec=5.856091145286419, CurrSamplesPerSec=5.300841387546959, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 118/5000 [21:58<15:01:32, 11.08s/it][2022-12-20 10:44:33,823] [INFO] [logging.py:68:log_dist] [Rank 0] step=120, skipped=2, lr=[7.676565519355727e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:44:33,824] [INFO] [timer.py:197:stop] 0/240, RunningAvgSamplesPerSec=5.856067446973333, CurrSamplesPerSec=5.284845388289936, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 119/5000 [22:09<15:00:47, 11.07s/it][2022-12-20 10:44:44,942] [INFO] [timer.py:197:stop] 0/242, RunningAvgSamplesPerSec=5.855774006960929, CurrSamplesPerSec=5.234855539021005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 120/5000 [22:20<15:01:44, 11.09s/it][2022-12-20 10:44:56,023] [INFO] [timer.py:197:stop] 0/244, RunningAvgSamplesPerSec=5.85572882945814, CurrSamplesPerSec=5.29646052568596, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 121/5000 [22:31<15:01:22, 11.08s/it][2022-12-20 10:45:07,121] [INFO] [timer.py:197:stop] 0/246, RunningAvgSamplesPerSec=5.8555346510128885, CurrSamplesPerSec=5.2956726851152425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 122/5000 [22:42<15:01:30, 11.09s/it][2022-12-20 10:45:18,219] [INFO] [timer.py:197:stop] 0/248, RunningAvgSamplesPerSec=5.855349067051514, CurrSamplesPerSec=5.267360469293578, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 123/5000 [22:53<15:01:33, 11.09s/it][2022-12-20 10:45:28,336] [INFO] [timer.py:197:stop] 0/250, RunningAvgSamplesPerSec=5.859359422146485, CurrSamplesPerSec=5.329779980639014, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▏         | 124/5000 [23:08<16:38:25, 12.29s/it][2022-12-20 10:45:39,416] [INFO] [timer.py:197:stop] 0/252, RunningAvgSamplesPerSec=5.859266752803543, CurrSamplesPerSec=5.281227903518494, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  2%|▎         | 125/5000 [23:19<16:08:40, 11.92s/it]
                                                     
{'loss': 0.0552, 'learning_rate': 7.756372548783638e-06, 'epoch': 3.05}

  2%|▎         | 125/5000 [23:19<16:08:40, 11.92s/it][2022-12-20 10:45:50,415] [INFO] [timer.py:197:stop] 0/254, RunningAvgSamplesPerSec=5.859546489254234, CurrSamplesPerSec=5.363369992439926, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 126/5000 [23:30<15:46:07, 11.65s/it][2022-12-20 10:46:01,492] [INFO] [timer.py:197:stop] 0/256, RunningAvgSamplesPerSec=5.8594099891047895, CurrSamplesPerSec=5.2820546887273645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 127/5000 [23:42<15:32:03, 11.48s/it][2022-12-20 10:46:12,621] [INFO] [timer.py:197:stop] 0/258, RunningAvgSamplesPerSec=5.859059312352249, CurrSamplesPerSec=5.238009878471587, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 128/5000 [23:53<15:23:10, 11.37s/it][2022-12-20 10:46:23,569] [INFO] [logging.py:68:log_dist] [Rank 0] step=130, skipped=2, lr=[7.807459757842952e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:46:23,570] [INFO] [timer.py:197:stop] 0/260, RunningAvgSamplesPerSec=5.8595003803902115, CurrSamplesPerSec=5.3875000737572565, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 129/5000 [24:04<15:13:38, 11.25s/it][2022-12-20 10:46:34,593] [INFO] [timer.py:197:stop] 0/262, RunningAvgSamplesPerSec=5.8595877003897385, CurrSamplesPerSec=5.353193637688269, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 130/5000 [24:15<15:07:46, 11.18s/it][2022-12-20 10:46:45,738] [INFO] [timer.py:197:stop] 0/264, RunningAvgSamplesPerSec=5.859174150175902, CurrSamplesPerSec=5.244390015578314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 131/5000 [24:26<15:05:15, 11.16s/it][2022-12-20 10:46:56,746] [INFO] [timer.py:197:stop] 0/266, RunningAvgSamplesPerSec=5.859336745050027, CurrSamplesPerSec=5.316046028976704, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 132/5000 [24:37<15:01:38, 11.11s/it][2022-12-20 10:47:07,747] [INFO] [timer.py:197:stop] 0/268, RunningAvgSamplesPerSec=5.859519025245256, CurrSamplesPerSec=5.32774367268126, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 133/5000 [24:48<14:58:29, 11.08s/it][2022-12-20 10:47:18,761] [INFO] [timer.py:197:stop] 0/270, RunningAvgSamplesPerSec=5.859646703327469, CurrSamplesPerSec=5.308115915204755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 134/5000 [24:59<14:57:51, 11.07s/it][2022-12-20 10:47:29,834] [INFO] [timer.py:197:stop] 0/272, RunningAvgSamplesPerSec=5.8595500566604315, CurrSamplesPerSec=5.296859550811445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 135/5000 [25:10<14:57:39, 11.07s/it][2022-12-20 10:47:40,859] [INFO] [timer.py:197:stop] 0/274, RunningAvgSamplesPerSec=5.859642292725243, CurrSamplesPerSec=5.334754898800506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 136/5000 [25:21<14:56:42, 11.06s/it][2022-12-20 10:47:51,954] [INFO] [timer.py:197:stop] 0/276, RunningAvgSamplesPerSec=5.8594461616665345, CurrSamplesPerSec=5.285049117885484, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 137/5000 [25:32<14:57:01, 11.07s/it][2022-12-20 10:48:03,026] [INFO] [timer.py:197:stop] 0/278, RunningAvgSamplesPerSec=5.859353314666908, CurrSamplesPerSec=5.294461497958766, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 138/5000 [25:43<14:56:51, 11.07s/it][2022-12-20 10:48:14,095] [INFO] [logging.py:68:log_dist] [Rank 0] step=140, skipped=2, lr=[7.928502661991142e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:48:14,097] [INFO] [timer.py:197:stop] 0/280, RunningAvgSamplesPerSec=5.859255511213661, CurrSamplesPerSec=5.291565294534488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 139/5000 [25:54<14:55:18, 11.05s/it][2022-12-20 10:48:25,076] [INFO] [timer.py:197:stop] 0/282, RunningAvgSamplesPerSec=5.859562081647636, CurrSamplesPerSec=5.331309768720038, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 140/5000 [26:05<14:55:08, 11.05s/it][2022-12-20 10:48:36,139] [INFO] [timer.py:197:stop] 0/284, RunningAvgSamplesPerSec=5.859493903894088, CurrSamplesPerSec=5.30991918159871, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 141/5000 [26:16<14:54:54, 11.05s/it][2022-12-20 10:48:47,179] [INFO] [timer.py:197:stop] 0/286, RunningAvgSamplesPerSec=5.8595132702829105, CurrSamplesPerSec=5.3180422173005795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 142/5000 [26:27<14:54:50, 11.05s/it][2022-12-20 10:48:58,245] [INFO] [timer.py:197:stop] 0/288, RunningAvgSamplesPerSec=5.859445798189604, CurrSamplesPerSec=5.309005740326129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 143/5000 [26:38<14:55:49, 11.07s/it][2022-12-20 10:49:09,356] [INFO] [timer.py:197:stop] 0/290, RunningAvgSamplesPerSec=5.8592499956896935, CurrSamplesPerSec=5.298814158061541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 144/5000 [26:49<14:56:05, 11.07s/it][2022-12-20 10:49:20,479] [INFO] [timer.py:197:stop] 0/292, RunningAvgSamplesPerSec=5.858969629537348, CurrSamplesPerSec=5.266786272063036, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 145/5000 [27:01<14:56:38, 11.08s/it][2022-12-20 10:49:31,537] [INFO] [timer.py:197:stop] 0/294, RunningAvgSamplesPerSec=5.858927506694944, CurrSamplesPerSec=5.305043085726473, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 146/5000 [27:12<14:55:49, 11.07s/it][2022-12-20 10:49:42,697] [INFO] [timer.py:197:stop] 0/296, RunningAvgSamplesPerSec=5.8585163077574585, CurrSamplesPerSec=5.214832350131838, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 147/5000 [27:23<14:58:07, 11.10s/it][2022-12-20 10:49:53,751] [INFO] [timer.py:197:stop] 0/298, RunningAvgSamplesPerSec=5.858507193735428, CurrSamplesPerSec=5.318793097784191, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 148/5000 [27:34<14:55:32, 11.07s/it][2022-12-20 10:50:04,862] [INFO] [logging.py:68:log_dist] [Rank 0] step=150, skipped=2, lr=[8.041073861170494e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:50:04,864] [INFO] [timer.py:197:stop] 0/300, RunningAvgSamplesPerSec=5.858272719945588, CurrSamplesPerSec=5.226106443907669, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 149/5000 [27:45<14:57:23, 11.10s/it][2022-12-20 10:50:16,072] [INFO] [timer.py:197:stop] 0/302, RunningAvgSamplesPerSec=5.857703425793956, CurrSamplesPerSec=5.184136644135131, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 150/5000 [27:56<14:59:06, 11.12s/it]
                                                     
{'loss': 0.0337, 'learning_rate': 8.051909672656425e-06, 'epoch': 3.65}

  3%|▎         | 150/5000 [27:56<14:59:06, 11.12s/it][2022-12-20 10:50:27,122] [INFO] [timer.py:197:stop] 0/304, RunningAvgSamplesPerSec=5.857704074093965, CurrSamplesPerSec=5.294869412707697, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 151/5000 [28:07<14:58:02, 11.11s/it][2022-12-20 10:50:38,208] [INFO] [timer.py:197:stop] 0/306, RunningAvgSamplesPerSec=5.85757452637261, CurrSamplesPerSec=5.29456592483238, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 152/5000 [28:19<15:04:52, 11.20s/it][2022-12-20 10:50:49,579] [INFO] [timer.py:197:stop] 0/308, RunningAvgSamplesPerSec=5.857381313093228, CurrSamplesPerSec=5.321076123599804, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 153/5000 [28:30<15:01:09, 11.16s/it][2022-12-20 10:51:00,677] [INFO] [timer.py:197:stop] 0/310, RunningAvgSamplesPerSec=5.857213402018397, CurrSamplesPerSec=5.281991080674629, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 154/5000 [28:41<14:58:07, 11.12s/it][2022-12-20 10:51:11,816] [INFO] [timer.py:197:stop] 0/312, RunningAvgSamplesPerSec=5.856907582706088, CurrSamplesPerSec=5.194828445646726, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 155/5000 [28:52<14:59:32, 11.14s/it][2022-12-20 10:51:22,842] [INFO] [timer.py:197:stop] 0/314, RunningAvgSamplesPerSec=5.8569910911672025, CurrSamplesPerSec=5.333290948145007, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 156/5000 [29:03<14:55:57, 11.10s/it][2022-12-20 10:51:33,873] [INFO] [timer.py:197:stop] 0/316, RunningAvgSamplesPerSec=5.857059922496942, CurrSamplesPerSec=5.305962715078064, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 157/5000 [29:14<14:57:07, 11.11s/it][2022-12-20 10:51:45,011] [INFO] [timer.py:197:stop] 0/318, RunningAvgSamplesPerSec=5.856760027964934, CurrSamplesPerSec=5.3188601246485785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 158/5000 [29:25<14:55:06, 11.09s/it][2022-12-20 10:51:56,055] [INFO] [logging.py:68:log_dist] [Rank 0] step=160, skipped=2, lr=[8.146282038785833e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:51:56,057] [INFO] [timer.py:197:stop] 0/320, RunningAvgSamplesPerSec=5.856831990627022, CurrSamplesPerSec=5.327325602348858, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 159/5000 [29:36<14:52:32, 11.06s/it][2022-12-20 10:52:07,126] [INFO] [timer.py:197:stop] 0/322, RunningAvgSamplesPerSec=5.856768721940556, CurrSamplesPerSec=5.2476935288845805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 160/5000 [29:47<14:53:09, 11.07s/it][2022-12-20 10:52:18,162] [INFO] [timer.py:197:stop] 0/324, RunningAvgSamplesPerSec=5.856816702737225, CurrSamplesPerSec=5.299424654049937, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 161/5000 [29:58<14:52:13, 11.06s/it][2022-12-20 10:52:29,190] [INFO] [timer.py:197:stop] 0/326, RunningAvgSamplesPerSec=5.8569415877813205, CurrSamplesPerSec=5.309800284119872, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 162/5000 [30:09<14:52:52, 11.07s/it][2022-12-20 10:52:40,312] [INFO] [timer.py:197:stop] 0/328, RunningAvgSamplesPerSec=5.856706143115146, CurrSamplesPerSec=5.28896319080173, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 163/5000 [30:20<14:52:29, 11.07s/it][2022-12-20 10:52:51,320] [INFO] [timer.py:197:stop] 0/330, RunningAvgSamplesPerSec=5.856845695302364, CurrSamplesPerSec=5.339719568871812, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 164/5000 [30:31<14:51:00, 11.05s/it][2022-12-20 10:53:01,480] [INFO] [timer.py:197:stop] 0/332, RunningAvgSamplesPerSec=5.859699949440496, CurrSamplesPerSec=6.216881783125628, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 10:53:12,666] [INFO] [timer.py:197:stop] 0/334, RunningAvgSamplesPerSec=5.85923408128199, CurrSamplesPerSec=5.300510420702654, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 165/5000 [30:48<16:59:43, 12.65s/it][2022-12-20 10:53:23,758] [INFO] [timer.py:197:stop] 0/336, RunningAvgSamplesPerSec=5.859141255186169, CurrSamplesPerSec=5.298936120484576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 166/5000 [30:59<16:21:43, 12.19s/it][2022-12-20 10:53:35,112] [INFO] [timer.py:197:stop] 0/338, RunningAvgSamplesPerSec=5.858161477486986, CurrSamplesPerSec=5.020684883584873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 167/5000 [31:10<16:01:27, 11.94s/it][2022-12-20 10:53:46,137] [INFO] [logging.py:68:log_dist] [Rank 0] step=170, skipped=2, lr=[8.245031542220927e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:53:46,138] [INFO] [timer.py:197:stop] 0/340, RunningAvgSamplesPerSec=5.858246941556209, CurrSamplesPerSec=5.317218241988234, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 168/5000 [31:21<15:39:16, 11.66s/it][2022-12-20 10:53:57,157] [INFO] [timer.py:197:stop] 0/342, RunningAvgSamplesPerSec=5.858397109574342, CurrSamplesPerSec=5.329777864186606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 169/5000 [31:32<15:23:29, 11.47s/it][2022-12-20 10:54:08,331] [INFO] [timer.py:197:stop] 0/344, RunningAvgSamplesPerSec=5.858001427038247, CurrSamplesPerSec=5.329987401129284, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 170/5000 [31:43<15:16:10, 11.38s/it][2022-12-20 10:54:19,406] [INFO] [timer.py:197:stop] 0/346, RunningAvgSamplesPerSec=5.85791697554979, CurrSamplesPerSec=5.2893283609704955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 171/5000 [31:55<15:08:35, 11.29s/it][2022-12-20 10:54:30,472] [INFO] [timer.py:197:stop] 0/348, RunningAvgSamplesPerSec=5.8578614260903406, CurrSamplesPerSec=5.314547713919809, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 172/5000 [32:06<15:03:00, 11.22s/it][2022-12-20 10:54:41,489] [INFO] [timer.py:197:stop] 0/350, RunningAvgSamplesPerSec=5.857997434605857, CurrSamplesPerSec=5.362084157781378, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 173/5000 [32:17<14:57:53, 11.16s/it][2022-12-20 10:54:52,552] [INFO] [timer.py:197:stop] 0/352, RunningAvgSamplesPerSec=5.8579533243042485, CurrSamplesPerSec=5.310998114963411, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  3%|▎         | 174/5000 [32:28<14:55:20, 11.13s/it][2022-12-20 10:55:03,688] [INFO] [timer.py:197:stop] 0/354, RunningAvgSamplesPerSec=5.857685995084753, CurrSamplesPerSec=5.231389742872569, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 175/5000 [32:39<14:55:15, 11.13s/it]
                                                     
{'loss': 0.0294, 'learning_rate': 8.310718700403308e-06, 'epoch': 4.27}

  4%|▎         | 175/5000 [32:39<14:55:15, 11.13s/it][2022-12-20 10:55:14,716] [INFO] [timer.py:197:stop] 0/356, RunningAvgSamplesPerSec=5.857755856558434, CurrSamplesPerSec=5.292608814894162, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 176/5000 [32:50<14:52:31, 11.10s/it][2022-12-20 10:55:25,783] [INFO] [timer.py:197:stop] 0/358, RunningAvgSamplesPerSec=5.857697877286784, CurrSamplesPerSec=5.310732700822355, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 177/5000 [33:01<14:51:31, 11.09s/it][2022-12-20 10:55:37,093] [INFO] [logging.py:68:log_dist] [Rank 0] step=180, skipped=2, lr=[8.338069703233054e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:55:37,095] [INFO] [timer.py:197:stop] 0/360, RunningAvgSamplesPerSec=5.856908170719769, CurrSamplesPerSec=5.303358380291688, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 178/5000 [33:12<14:56:40, 11.16s/it][2022-12-20 10:55:48,125] [INFO] [timer.py:197:stop] 0/362, RunningAvgSamplesPerSec=5.856981412453173, CurrSamplesPerSec=5.313350595351179, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 179/5000 [33:23<14:53:25, 11.12s/it][2022-12-20 10:55:59,171] [INFO] [timer.py:197:stop] 0/364, RunningAvgSamplesPerSec=5.8569916804092825, CurrSamplesPerSec=5.278928067625543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 180/5000 [33:34<14:51:29, 11.10s/it][2022-12-20 10:56:10,232] [INFO] [timer.py:197:stop] 0/366, RunningAvgSamplesPerSec=5.856962127677191, CurrSamplesPerSec=5.323708028979358, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 181/5000 [33:45<14:50:24, 11.09s/it][2022-12-20 10:56:20,693] [INFO] [stage_1_and_2.py:1765:step] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 32768.0, reducing to 16384.0
[2022-12-20 10:56:20,695] [INFO] [timer.py:197:stop] 0/368, RunningAvgSamplesPerSec=5.858725857501175, CurrSamplesPerSec=5.898000393734004, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 182/5000 [33:56<14:35:11, 10.90s/it][2022-12-20 10:56:31,855] [INFO] [timer.py:197:stop] 0/370, RunningAvgSamplesPerSec=5.858439375032678, CurrSamplesPerSec=5.286197497603605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 183/5000 [34:07<14:41:19, 10.98s/it][2022-12-20 10:56:42,886] [INFO] [timer.py:197:stop] 0/372, RunningAvgSamplesPerSec=5.8584900089089516, CurrSamplesPerSec=5.293871354697475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 184/5000 [34:18<14:42:25, 10.99s/it][2022-12-20 10:56:54,119] [INFO] [timer.py:197:stop] 0/374, RunningAvgSamplesPerSec=5.8579652316386275, CurrSamplesPerSec=5.159452261553984, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 185/5000 [34:29<14:48:06, 11.07s/it][2022-12-20 10:57:05,208] [INFO] [timer.py:197:stop] 0/376, RunningAvgSamplesPerSec=5.8580714283355935, CurrSamplesPerSec=5.331543993132671, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 186/5000 [34:40<14:48:21, 11.07s/it][2022-12-20 10:57:16,226] [INFO] [timer.py:197:stop] 0/378, RunningAvgSamplesPerSec=5.858158175076429, CurrSamplesPerSec=5.318725440304163, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▎         | 187/5000 [34:51<14:46:51, 11.06s/it][2022-12-20 10:57:27,377] [INFO] [logging.py:68:log_dist] [Rank 0] step=190, skipped=3, lr=[8.417439256037237e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:57:27,379] [INFO] [timer.py:197:stop] 0/380, RunningAvgSamplesPerSec=5.8578591703401965, CurrSamplesPerSec=5.216399640030462, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 188/5000 [35:02<14:49:00, 11.08s/it][2022-12-20 10:57:38,422] [INFO] [timer.py:197:stop] 0/382, RunningAvgSamplesPerSec=5.857874604658253, CurrSamplesPerSec=5.299995316718187, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 189/5000 [35:14<14:47:49, 11.07s/it][2022-12-20 10:57:49,390] [INFO] [timer.py:197:stop] 0/384, RunningAvgSamplesPerSec=5.858101129874421, CurrSamplesPerSec=5.347631324183106, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 190/5000 [35:24<14:45:07, 11.04s/it][2022-12-20 10:58:00,611] [INFO] [timer.py:197:stop] 0/386, RunningAvgSamplesPerSec=5.857616373973327, CurrSamplesPerSec=5.315675897334509, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 191/5000 [35:36<14:49:16, 11.10s/it][2022-12-20 10:58:11,772] [INFO] [timer.py:197:stop] 0/388, RunningAvgSamplesPerSec=5.8573107501256825, CurrSamplesPerSec=5.256945203310765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 192/5000 [35:47<14:50:40, 11.11s/it][2022-12-20 10:58:23,176] [INFO] [timer.py:197:stop] 0/390, RunningAvgSamplesPerSec=5.856333069336772, CurrSamplesPerSec=5.016195199746577, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 193/5000 [35:58<14:57:26, 11.20s/it][2022-12-20 10:58:34,164] [INFO] [timer.py:197:stop] 0/392, RunningAvgSamplesPerSec=5.856513853733961, CurrSamplesPerSec=5.348452178281888, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 194/5000 [36:09<14:52:08, 11.14s/it][2022-12-20 10:58:45,201] [INFO] [timer.py:197:stop] 0/394, RunningAvgSamplesPerSec=5.856601694955397, CurrSamplesPerSec=5.349704188874938, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 195/5000 [36:20<14:49:30, 11.11s/it][2022-12-20 10:58:56,341] [INFO] [timer.py:197:stop] 0/396, RunningAvgSamplesPerSec=5.856362573983961, CurrSamplesPerSec=5.312952446806043, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 196/5000 [36:31<14:50:06, 11.12s/it][2022-12-20 10:59:07,435] [INFO] [timer.py:197:stop] 0/398, RunningAvgSamplesPerSec=5.856251394972031, CurrSamplesPerSec=5.269686855866364, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 197/5000 [36:43<14:49:21, 11.11s/it][2022-12-20 10:59:18,607] [INFO] [logging.py:68:log_dist] [Rank 0] step=200, skipped=3, lr=[8.501266121799902e-06], mom=[[0.9, 0.999]]
[2022-12-20 10:59:18,608] [INFO] [timer.py:197:stop] 0/400, RunningAvgSamplesPerSec=5.8559254259496125, CurrSamplesPerSec=5.204242362978514, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 198/5000 [36:54<14:50:41, 11.13s/it][2022-12-20 10:59:29,643] [INFO] [timer.py:197:stop] 0/402, RunningAvgSamplesPerSec=5.856018646908684, CurrSamplesPerSec=5.3040401409850535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 199/5000 [37:05<14:48:14, 11.10s/it][2022-12-20 10:59:40,704] [INFO] [timer.py:197:stop] 0/404, RunningAvgSamplesPerSec=5.855996056015223, CurrSamplesPerSec=5.310881690925986, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 200/5000 [37:16<14:47:06, 11.09s/it]
                                                     
{'loss': 0.0227, 'learning_rate': 8.517519915806749e-06, 'epoch': 4.87}

  4%|▍         | 200/5000 [37:16<14:47:06, 11.09s/it][2022-12-20 10:59:51,744] [INFO] [timer.py:197:stop] 0/406, RunningAvgSamplesPerSec=5.85603818723615, CurrSamplesPerSec=5.333395852715832, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 201/5000 [37:27<14:45:44, 11.07s/it][2022-12-20 11:00:02,761] [INFO] [timer.py:197:stop] 0/408, RunningAvgSamplesPerSec=5.8561320418621685, CurrSamplesPerSec=5.341774822987532, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 202/5000 [37:38<14:44:11, 11.06s/it][2022-12-20 11:00:13,816] [INFO] [timer.py:197:stop] 0/410, RunningAvgSamplesPerSec=5.856125225333127, CurrSamplesPerSec=5.299679731560478, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 203/5000 [37:49<14:43:57, 11.06s/it][2022-12-20 11:00:24,916] [INFO] [timer.py:197:stop] 0/412, RunningAvgSamplesPerSec=5.8559978335529435, CurrSamplesPerSec=5.306314292757185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 204/5000 [38:00<14:44:50, 11.07s/it][2022-12-20 11:00:35,941] [INFO] [timer.py:197:stop] 0/414, RunningAvgSamplesPerSec=5.856073360342333, CurrSamplesPerSec=5.333101282679855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 205/5000 [38:11<14:43:33, 11.06s/it][2022-12-20 11:00:46,090] [INFO] [timer.py:197:stop] 0/416, RunningAvgSamplesPerSec=5.85838926345125, CurrSamplesPerSec=5.288194247307051, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 206/5000 [38:26<16:20:16, 12.27s/it][2022-12-20 11:00:57,094] [INFO] [timer.py:197:stop] 0/418, RunningAvgSamplesPerSec=5.858502204332879, CurrSamplesPerSec=5.341394722977217, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 207/5000 [38:37<15:48:53, 11.88s/it][2022-12-20 11:01:08,083] [INFO] [logging.py:68:log_dist] [Rank 0] step=210, skipped=3, lr=[8.580941402595085e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:01:08,085] [INFO] [timer.py:197:stop] 0/420, RunningAvgSamplesPerSec=5.858647483153495, CurrSamplesPerSec=5.320730813281248, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 208/5000 [38:48<15:28:59, 11.63s/it][2022-12-20 11:01:19,179] [INFO] [timer.py:197:stop] 0/422, RunningAvgSamplesPerSec=5.8585707880202, CurrSamplesPerSec=5.301717466369746, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 209/5000 [38:59<15:14:49, 11.46s/it][2022-12-20 11:01:30,227] [INFO] [timer.py:197:stop] 0/424, RunningAvgSamplesPerSec=5.858571872298595, CurrSamplesPerSec=5.2891620272091044, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 210/5000 [39:10<15:04:53, 11.33s/it][2022-12-20 11:01:41,577] [INFO] [timer.py:197:stop] 0/426, RunningAvgSamplesPerSec=5.857808053457574, CurrSamplesPerSec=5.038490295298745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 211/5000 [39:22<15:05:48, 11.35s/it][2022-12-20 11:01:52,605] [INFO] [timer.py:197:stop] 0/428, RunningAvgSamplesPerSec=5.857857597228842, CurrSamplesPerSec=5.333948842185247, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 212/5000 [39:33<14:56:38, 11.24s/it][2022-12-20 11:02:03,602] [INFO] [timer.py:197:stop] 0/430, RunningAvgSamplesPerSec=5.85798763622133, CurrSamplesPerSec=5.314254801331524, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 213/5000 [39:44<14:54:26, 11.21s/it][2022-12-20 11:02:14,794] [INFO] [timer.py:197:stop] 0/432, RunningAvgSamplesPerSec=5.8576770378946765, CurrSamplesPerSec=5.294506818715968, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 214/5000 [39:55<14:50:09, 11.16s/it][2022-12-20 11:02:25,769] [INFO] [timer.py:197:stop] 0/434, RunningAvgSamplesPerSec=5.857859291154519, CurrSamplesPerSec=5.336133731134219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 215/5000 [40:06<14:47:02, 11.12s/it][2022-12-20 11:02:36,997] [INFO] [timer.py:197:stop] 0/436, RunningAvgSamplesPerSec=5.857442258695179, CurrSamplesPerSec=5.180132208690689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 216/5000 [40:17<14:52:42, 11.20s/it][2022-12-20 11:02:48,171] [INFO] [timer.py:197:stop] 0/438, RunningAvgSamplesPerSec=5.857509015594741, CurrSamplesPerSec=5.338328266767564, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 217/5000 [40:28<14:48:33, 11.15s/it][2022-12-20 11:02:59,238] [INFO] [logging.py:68:log_dist] [Rank 0] step=220, skipped=3, lr=[8.656856986535236e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:02:59,239] [INFO] [timer.py:197:stop] 0/440, RunningAvgSamplesPerSec=5.857460152729154, CurrSamplesPerSec=5.304437793387662, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 218/5000 [40:39<14:46:09, 11.12s/it][2022-12-20 11:03:10,348] [INFO] [timer.py:197:stop] 0/442, RunningAvgSamplesPerSec=5.8573150606370525, CurrSamplesPerSec=5.256851108549703, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 219/5000 [40:50<14:46:20, 11.12s/it][2022-12-20 11:03:21,411] [INFO] [timer.py:197:stop] 0/444, RunningAvgSamplesPerSec=5.857281410548134, CurrSamplesPerSec=5.317874494135658, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 220/5000 [41:01<14:43:10, 11.09s/it][2022-12-20 11:03:32,445] [INFO] [timer.py:197:stop] 0/446, RunningAvgSamplesPerSec=5.857317767980774, CurrSamplesPerSec=5.287316591377659, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 221/5000 [41:13<14:42:44, 11.08s/it][2022-12-20 11:03:43,459] [INFO] [timer.py:197:stop] 0/448, RunningAvgSamplesPerSec=5.857403312902798, CurrSamplesPerSec=5.342278092292819, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 222/5000 [41:23<14:39:45, 11.05s/it][2022-12-20 11:03:54,457] [INFO] [timer.py:197:stop] 0/450, RunningAvgSamplesPerSec=5.857523756620263, CurrSamplesPerSec=5.313162976452456, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 223/5000 [41:35<14:40:31, 11.06s/it][2022-12-20 11:04:05,538] [INFO] [timer.py:197:stop] 0/452, RunningAvgSamplesPerSec=5.857458122232465, CurrSamplesPerSec=5.321594278622332, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 224/5000 [41:46<14:38:57, 11.04s/it][2022-12-20 11:04:16,552] [INFO] [timer.py:197:stop] 0/454, RunningAvgSamplesPerSec=5.8575428426544205, CurrSamplesPerSec=5.308034884132045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  4%|▍         | 225/5000 [41:57<14:38:04, 11.03s/it]
                                                     
{'loss': 0.019, 'learning_rate': 8.707944195594548e-06, 'epoch': 5.48}

  4%|▍         | 225/5000 [41:57<14:38:04, 11.03s/it][2022-12-20 11:04:27,612] [INFO] [timer.py:197:stop] 0/456, RunningAvgSamplesPerSec=5.857515961725978, CurrSamplesPerSec=5.266657105271161, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 226/5000 [42:08<14:39:49, 11.06s/it][2022-12-20 11:04:38,675] [INFO] [timer.py:197:stop] 0/458, RunningAvgSamplesPerSec=5.857482686703352, CurrSamplesPerSec=5.311742803133514, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 227/5000 [42:19<14:39:49, 11.06s/it][2022-12-20 11:04:50,185] [INFO] [logging.py:68:log_dist] [Rank 0] step=230, skipped=3, lr=[8.729351765332922e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:04:50,186] [INFO] [timer.py:197:stop] 0/460, RunningAvgSamplesPerSec=5.856400996898825, CurrSamplesPerSec=4.945712972225567, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 228/5000 [42:30<14:49:22, 11.18s/it][2022-12-20 11:05:01,182] [INFO] [timer.py:197:stop] 0/462, RunningAvgSamplesPerSec=5.856531286542138, CurrSamplesPerSec=5.334872159839005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 229/5000 [42:41<14:45:15, 11.13s/it][2022-12-20 11:05:12,228] [INFO] [timer.py:197:stop] 0/464, RunningAvgSamplesPerSec=5.856543413203302, CurrSamplesPerSec=5.31035784591749, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 230/5000 [42:52<14:43:18, 11.11s/it][2022-12-20 11:05:23,291] [INFO] [timer.py:197:stop] 0/466, RunningAvgSamplesPerSec=5.856513544319843, CurrSamplesPerSec=5.3065528299168445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 231/5000 [43:03<14:40:41, 11.08s/it][2022-12-20 11:05:34,306] [INFO] [timer.py:197:stop] 0/468, RunningAvgSamplesPerSec=5.8565947664098665, CurrSamplesPerSec=5.301360006316553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 232/5000 [43:14<14:39:34, 11.07s/it][2022-12-20 11:05:45,454] [INFO] [timer.py:197:stop] 0/470, RunningAvgSamplesPerSec=5.856370349920282, CurrSamplesPerSec=5.208688015146199, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 233/5000 [43:25<14:41:13, 11.09s/it][2022-12-20 11:05:56,459] [INFO] [timer.py:197:stop] 0/472, RunningAvgSamplesPerSec=5.856475625555472, CurrSamplesPerSec=5.330831006897113, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 234/5000 [43:37<14:39:37, 11.07s/it][2022-12-20 11:06:07,580] [INFO] [timer.py:197:stop] 0/474, RunningAvgSamplesPerSec=5.856317480650445, CurrSamplesPerSec=5.253137777514068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 235/5000 [43:48<14:39:43, 11.08s/it][2022-12-20 11:06:18,618] [INFO] [timer.py:197:stop] 0/476, RunningAvgSamplesPerSec=5.8563479932566604, CurrSamplesPerSec=5.29463610200572, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 236/5000 [43:59<14:39:58, 11.08s/it][2022-12-20 11:06:29,704] [INFO] [timer.py:197:stop] 0/478, RunningAvgSamplesPerSec=5.856267810786452, CurrSamplesPerSec=5.302561363806155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 237/5000 [44:10<14:40:42, 11.09s/it][2022-12-20 11:06:40,818] [INFO] [logging.py:68:log_dist] [Rank 0] step=240, skipped=3, lr=[8.798720779389776e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:06:40,820] [INFO] [timer.py:197:stop] 0/480, RunningAvgSamplesPerSec=5.856121799855575, CurrSamplesPerSec=5.30817154677762, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 238/5000 [44:21<14:38:38, 11.07s/it][2022-12-20 11:06:51,846] [INFO] [timer.py:197:stop] 0/482, RunningAvgSamplesPerSec=5.856179469658789, CurrSamplesPerSec=5.299115622564239, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 239/5000 [44:32<14:38:21, 11.07s/it][2022-12-20 11:07:02,837] [INFO] [timer.py:197:stop] 0/484, RunningAvgSamplesPerSec=5.856316533151863, CurrSamplesPerSec=5.366641674961138, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 240/5000 [44:43<14:35:42, 11.04s/it][2022-12-20 11:07:13,882] [INFO] [timer.py:197:stop] 0/486, RunningAvgSamplesPerSec=5.8563316227588045, CurrSamplesPerSec=5.295652208565427, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 241/5000 [44:54<14:35:49, 11.04s/it][2022-12-20 11:07:25,193] [INFO] [timer.py:197:stop] 0/488, RunningAvgSamplesPerSec=5.855758352846833, CurrSamplesPerSec=5.076731661178443, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 242/5000 [45:05<14:41:59, 11.12s/it][2022-12-20 11:07:36,227] [INFO] [timer.py:197:stop] 0/490, RunningAvgSamplesPerSec=5.8558000181211005, CurrSamplesPerSec=5.309312356065239, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 243/5000 [45:16<14:40:02, 11.10s/it][2022-12-20 11:07:47,480] [INFO] [timer.py:197:stop] 0/492, RunningAvgSamplesPerSec=5.8553583240988525, CurrSamplesPerSec=5.133434769661231, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 244/5000 [45:28<14:43:26, 11.15s/it][2022-12-20 11:07:58,510] [INFO] [timer.py:197:stop] 0/494, RunningAvgSamplesPerSec=5.855407527447282, CurrSamplesPerSec=5.322726933656451, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 245/5000 [45:39<14:41:02, 11.12s/it][2022-12-20 11:08:09,584] [INFO] [timer.py:197:stop] 0/496, RunningAvgSamplesPerSec=5.85536191989137, CurrSamplesPerSec=5.303040508771777, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 246/5000 [45:50<14:45:35, 11.18s/it][2022-12-20 11:08:19,971] [INFO] [timer.py:197:stop] 0/498, RunningAvgSamplesPerSec=5.856779091342151, CurrSamplesPerSec=6.255737984265681, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 11:08:30,961] [INFO] [logging.py:68:log_dist] [Rank 0] step=250, skipped=3, lr=[8.865222471593567e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:08:30,963] [INFO] [timer.py:197:stop] 0/500, RunningAvgSamplesPerSec=5.8568972081963535, CurrSamplesPerSec=5.353461604795685, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 247/5000 [46:06<16:43:50, 12.67s/it][2022-12-20 11:08:42,030] [INFO] [timer.py:197:stop] 0/502, RunningAvgSamplesPerSec=5.856860551186875, CurrSamplesPerSec=5.2955533800974415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 248/5000 [46:17<16:05:30, 12.19s/it][2022-12-20 11:08:53,095] [INFO] [timer.py:197:stop] 0/504, RunningAvgSamplesPerSec=5.856860623876325, CurrSamplesPerSec=5.289004249086605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▍         | 249/5000 [46:28<15:38:32, 11.85s/it][2022-12-20 11:09:04,177] [INFO] [timer.py:197:stop] 0/506, RunningAvgSamplesPerSec=5.856791544253622, CurrSamplesPerSec=5.300416015852075, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 250/5000 [46:39<15:20:03, 11.62s/it]
                                                     
{'loss': 0.0167, 'learning_rate': 8.884648606022435e-06, 'epoch': 6.1}

  5%|▌         | 250/5000 [46:39<15:20:03, 11.62s/it][2022-12-20 11:09:15,191] [INFO] [timer.py:197:stop] 0/508, RunningAvgSamplesPerSec=5.856871770448555, CurrSamplesPerSec=5.3041231462621505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 251/5000 [46:50<15:05:25, 11.44s/it][2022-12-20 11:09:26,240] [INFO] [timer.py:197:stop] 0/510, RunningAvgSamplesPerSec=5.856874561661358, CurrSamplesPerSec=5.322757541236206, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 252/5000 [47:01<14:55:57, 11.32s/it][2022-12-20 11:09:37,508] [INFO] [timer.py:197:stop] 0/512, RunningAvgSamplesPerSec=5.856414790542563, CurrSamplesPerSec=5.328918723362896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 253/5000 [47:13<14:54:29, 11.31s/it][2022-12-20 11:09:48,555] [INFO] [timer.py:197:stop] 0/514, RunningAvgSamplesPerSec=5.856424191686396, CurrSamplesPerSec=5.321866477018601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 254/5000 [47:24<14:48:09, 11.23s/it][2022-12-20 11:09:59,747] [INFO] [timer.py:197:stop] 0/516, RunningAvgSamplesPerSec=5.856224162499738, CurrSamplesPerSec=5.292336470856366, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 255/5000 [47:35<14:47:06, 11.22s/it][2022-12-20 11:10:10,725] [INFO] [timer.py:197:stop] 0/518, RunningAvgSamplesPerSec=5.856376042762672, CurrSamplesPerSec=5.348643576649328, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 256/5000 [47:46<14:41:14, 11.15s/it][2022-12-20 11:10:21,803] [INFO] [logging.py:68:log_dist] [Rank 0] step=260, skipped=3, lr=[8.929084500604405e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:10:21,804] [INFO] [timer.py:197:stop] 0/520, RunningAvgSamplesPerSec=5.856317346389712, CurrSamplesPerSec=5.341057822724815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 257/5000 [47:57<14:39:29, 11.13s/it][2022-12-20 11:10:32,854] [INFO] [timer.py:197:stop] 0/522, RunningAvgSamplesPerSec=5.856324090875313, CurrSamplesPerSec=5.304798184623223, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 258/5000 [48:08<14:37:30, 11.10s/it][2022-12-20 11:10:43,877] [INFO] [timer.py:197:stop] 0/524, RunningAvgSamplesPerSec=5.856383614193661, CurrSamplesPerSec=5.304977874405388, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 259/5000 [48:19<14:35:25, 11.08s/it][2022-12-20 11:10:54,964] [INFO] [timer.py:197:stop] 0/526, RunningAvgSamplesPerSec=5.856347032708378, CurrSamplesPerSec=5.302818419447315, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 260/5000 [48:30<14:35:26, 11.08s/it][2022-12-20 11:11:05,980] [INFO] [timer.py:197:stop] 0/528, RunningAvgSamplesPerSec=5.8564512371474065, CurrSamplesPerSec=5.32300705891386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 261/5000 [48:41<14:33:41, 11.06s/it][2022-12-20 11:11:17,043] [INFO] [timer.py:197:stop] 0/530, RunningAvgSamplesPerSec=5.856494233099665, CurrSamplesPerSec=5.31066776981019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 262/5000 [48:52<14:33:33, 11.06s/it][2022-12-20 11:11:28,089] [INFO] [timer.py:197:stop] 0/532, RunningAvgSamplesPerSec=5.856549021537043, CurrSamplesPerSec=5.301390578050552, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 263/5000 [49:03<14:32:58, 11.06s/it][2022-12-20 11:11:39,119] [INFO] [timer.py:197:stop] 0/534, RunningAvgSamplesPerSec=5.856590201564322, CurrSamplesPerSec=5.31503597332838, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 264/5000 [49:14<14:32:07, 11.05s/it][2022-12-20 11:11:50,147] [INFO] [timer.py:197:stop] 0/536, RunningAvgSamplesPerSec=5.856657464986529, CurrSamplesPerSec=5.322892218769101, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 265/5000 [49:25<14:31:27, 11.04s/it][2022-12-20 11:12:01,201] [INFO] [timer.py:197:stop] 0/538, RunningAvgSamplesPerSec=5.856683939251671, CurrSamplesPerSec=5.3007980517756375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 266/5000 [49:36<14:31:32, 11.05s/it][2022-12-20 11:12:12,233] [INFO] [logging.py:68:log_dist] [Rank 0] step=270, skipped=3, lr=[8.990508443836999e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:12:12,234] [INFO] [timer.py:197:stop] 0/540, RunningAvgSamplesPerSec=5.856720851769594, CurrSamplesPerSec=5.299691868756212, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 267/5000 [49:47<14:31:02, 11.04s/it][2022-12-20 11:12:23,393] [INFO] [timer.py:197:stop] 0/542, RunningAvgSamplesPerSec=5.856506426065606, CurrSamplesPerSec=5.234120617363708, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 268/5000 [49:58<14:33:38, 11.08s/it][2022-12-20 11:12:34,373] [INFO] [timer.py:197:stop] 0/544, RunningAvgSamplesPerSec=5.856680353704604, CurrSamplesPerSec=5.380234443014448, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 269/5000 [50:09<14:31:07, 11.05s/it][2022-12-20 11:12:45,435] [INFO] [timer.py:197:stop] 0/546, RunningAvgSamplesPerSec=5.856657579318055, CurrSamplesPerSec=5.30904165033851, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 270/5000 [50:21<14:31:16, 11.05s/it][2022-12-20 11:12:56,564] [INFO] [timer.py:197:stop] 0/548, RunningAvgSamplesPerSec=5.85650463837972, CurrSamplesPerSec=5.302840418031464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 271/5000 [50:32<14:32:55, 11.08s/it][2022-12-20 11:13:07,651] [INFO] [timer.py:197:stop] 0/550, RunningAvgSamplesPerSec=5.856438815867202, CurrSamplesPerSec=5.280224594704823, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 272/5000 [50:43<14:33:00, 11.08s/it][2022-12-20 11:13:18,668] [INFO] [timer.py:197:stop] 0/552, RunningAvgSamplesPerSec=5.856507149808677, CurrSamplesPerSec=5.3150538638466225, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 273/5000 [50:54<14:31:21, 11.06s/it][2022-12-20 11:13:29,734] [INFO] [timer.py:197:stop] 0/554, RunningAvgSamplesPerSec=5.856479431390849, CurrSamplesPerSec=5.300391106928927, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  5%|▌         | 274/5000 [51:05<14:31:19, 11.06s/it][2022-12-20 11:13:40,780] [INFO] [timer.py:197:stop] 0/556, RunningAvgSamplesPerSec=5.856511563704863, CurrSamplesPerSec=5.316737584769868, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 275/5000 [51:16<14:30:45, 11.06s/it]
                                                     
{'loss': 0.012, 'learning_rate': 9.038013352913754e-06, 'epoch': 6.7}

  6%|▌         | 275/5000 [51:16<14:30:45, 11.06s/it][2022-12-20 11:13:51,939] [INFO] [timer.py:197:stop] 0/558, RunningAvgSamplesPerSec=5.856306057659301, CurrSamplesPerSec=5.24733490534503, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 276/5000 [51:27<14:32:58, 11.09s/it][2022-12-20 11:14:02,918] [INFO] [logging.py:68:log_dist] [Rank 0] step=280, skipped=3, lr=[9.049673635277508e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:14:02,919] [INFO] [timer.py:197:stop] 0/560, RunningAvgSamplesPerSec=5.856445101055436, CurrSamplesPerSec=5.334963766985412, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 277/5000 [51:38<14:30:14, 11.06s/it][2022-12-20 11:14:13,969] [INFO] [timer.py:197:stop] 0/562, RunningAvgSamplesPerSec=5.85647555108703, CurrSamplesPerSec=5.32000785291162, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 278/5000 [51:49<14:29:56, 11.05s/it][2022-12-20 11:14:24,927] [INFO] [timer.py:197:stop] 0/564, RunningAvgSamplesPerSec=5.856657901549344, CurrSamplesPerSec=5.361128698465517, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 279/5000 [52:00<14:27:28, 11.02s/it][2022-12-20 11:14:35,975] [INFO] [timer.py:197:stop] 0/566, RunningAvgSamplesPerSec=5.856663759301973, CurrSamplesPerSec=5.304321866419407, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 280/5000 [52:11<14:27:51, 11.03s/it][2022-12-20 11:14:47,035] [INFO] [timer.py:197:stop] 0/568, RunningAvgSamplesPerSec=5.8566509641531574, CurrSamplesPerSec=5.282486681980968, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 281/5000 [52:22<14:28:20, 11.04s/it][2022-12-20 11:14:58,105] [INFO] [timer.py:197:stop] 0/570, RunningAvgSamplesPerSec=5.856651812690238, CurrSamplesPerSec=5.309971279698234, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 282/5000 [52:33<14:28:49, 11.05s/it][2022-12-20 11:15:09,180] [INFO] [timer.py:197:stop] 0/572, RunningAvgSamplesPerSec=5.8566074739083, CurrSamplesPerSec=5.311116435595688, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 283/5000 [52:44<14:29:15, 11.06s/it][2022-12-20 11:15:20,196] [INFO] [timer.py:197:stop] 0/574, RunningAvgSamplesPerSec=5.856676326948201, CurrSamplesPerSec=5.34164726661786, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 284/5000 [52:55<14:28:06, 11.04s/it][2022-12-20 11:15:31,281] [INFO] [timer.py:197:stop] 0/576, RunningAvgSamplesPerSec=5.856615505885273, CurrSamplesPerSec=5.2967928682539975, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 285/5000 [53:06<14:28:51, 11.06s/it][2022-12-20 11:15:42,721] [INFO] [timer.py:197:stop] 0/578, RunningAvgSamplesPerSec=5.855892201366843, CurrSamplesPerSec=4.993576103172258, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 286/5000 [53:18<14:37:43, 11.17s/it][2022-12-20 11:15:53,786] [INFO] [logging.py:68:log_dist] [Rank 0] step=290, skipped=3, lr=[9.106740322364802e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:15:53,788] [INFO] [timer.py:197:stop] 0/580, RunningAvgSamplesPerSec=5.855868479580955, CurrSamplesPerSec=5.309439002900382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 287/5000 [53:29<14:35:04, 11.14s/it][2022-12-20 11:16:03,958] [INFO] [timer.py:197:stop] 0/582, RunningAvgSamplesPerSec=5.857513456462526, CurrSamplesPerSec=5.291682124985008, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 288/5000 [53:44<16:08:49, 12.34s/it][2022-12-20 11:16:15,087] [INFO] [timer.py:197:stop] 0/584, RunningAvgSamplesPerSec=5.857368207908397, CurrSamplesPerSec=5.239125429877479, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 289/5000 [53:55<15:39:33, 11.97s/it][2022-12-20 11:16:26,060] [INFO] [timer.py:197:stop] 0/586, RunningAvgSamplesPerSec=5.857509700814061, CurrSamplesPerSec=5.353206234744181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 290/5000 [54:06<15:16:58, 11.68s/it][2022-12-20 11:16:37,132] [INFO] [timer.py:197:stop] 0/588, RunningAvgSamplesPerSec=5.857482579653141, CurrSamplesPerSec=5.302065549366416, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 291/5000 [54:17<15:02:19, 11.50s/it][2022-12-20 11:16:48,183] [INFO] [timer.py:197:stop] 0/590, RunningAvgSamplesPerSec=5.857481156572177, CurrSamplesPerSec=5.317685923563409, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 292/5000 [54:28<14:51:14, 11.36s/it][2022-12-20 11:16:59,230] [INFO] [timer.py:197:stop] 0/592, RunningAvgSamplesPerSec=5.8574855000183295, CurrSamplesPerSec=5.306330026737682, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 293/5000 [54:39<14:43:51, 11.27s/it][2022-12-20 11:17:10,353] [INFO] [timer.py:197:stop] 0/594, RunningAvgSamplesPerSec=5.857354490097212, CurrSamplesPerSec=5.245556258214396, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 294/5000 [54:50<14:40:30, 11.23s/it][2022-12-20 11:17:21,376] [INFO] [timer.py:197:stop] 0/596, RunningAvgSamplesPerSec=5.857401742468768, CurrSamplesPerSec=5.340350576753302, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 295/5000 [55:01<14:36:20, 11.18s/it][2022-12-20 11:17:32,441] [INFO] [timer.py:197:stop] 0/598, RunningAvgSamplesPerSec=5.857373673423125, CurrSamplesPerSec=5.333214868421894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 296/5000 [55:13<14:33:23, 11.14s/it][2022-12-20 11:17:43,537] [INFO] [logging.py:68:log_dist] [Rank 0] step=300, skipped=3, lr=[9.161852281961698e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:17:43,539] [INFO] [timer.py:197:stop] 0/600, RunningAvgSamplesPerSec=5.857309028483266, CurrSamplesPerSec=5.3088608452109005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 297/5000 [55:24<14:31:18, 11.12s/it][2022-12-20 11:17:54,579] [INFO] [timer.py:197:stop] 0/602, RunningAvgSamplesPerSec=5.8573229371700215, CurrSamplesPerSec=5.3144723784313666, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 298/5000 [55:35<14:29:23, 11.09s/it][2022-12-20 11:18:05,642] [INFO] [timer.py:197:stop] 0/604, RunningAvgSamplesPerSec=5.857298101174689, CurrSamplesPerSec=5.296603281566072, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 299/5000 [55:46<14:29:25, 11.10s/it][2022-12-20 11:18:16,678] [INFO] [timer.py:197:stop] 0/606, RunningAvgSamplesPerSec=5.857336187920247, CurrSamplesPerSec=5.35612137778279, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 300/5000 [55:57<14:25:52, 11.05s/it]
                                                     
{'loss': 0.0109, 'learning_rate': 9.178024397233219e-06, 'epoch': 7.31}

  6%|▌         | 300/5000 [55:57<14:25:52, 11.05s/it][2022-12-20 11:18:27,664] [INFO] [timer.py:197:stop] 0/608, RunningAvgSamplesPerSec=5.8574499310157755, CurrSamplesPerSec=5.327944589708572, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 301/5000 [56:08<14:25:41, 11.05s/it][2022-12-20 11:18:38,819] [INFO] [timer.py:197:stop] 0/610, RunningAvgSamplesPerSec=5.857262028280245, CurrSamplesPerSec=5.238492148939464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 302/5000 [56:19<14:27:29, 11.08s/it][2022-12-20 11:18:49,892] [INFO] [timer.py:197:stop] 0/612, RunningAvgSamplesPerSec=5.857219948841105, CurrSamplesPerSec=5.295356987748589, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 303/5000 [56:30<14:26:54, 11.07s/it][2022-12-20 11:19:00,997] [INFO] [timer.py:197:stop] 0/614, RunningAvgSamplesPerSec=5.857124149682448, CurrSamplesPerSec=5.258293372447988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 304/5000 [56:41<14:30:31, 11.12s/it][2022-12-20 11:19:12,177] [INFO] [timer.py:197:stop] 0/616, RunningAvgSamplesPerSec=5.856898392161057, CurrSamplesPerSec=5.307406872255252, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 305/5000 [56:52<14:29:06, 11.11s/it][2022-12-20 11:19:23,261] [INFO] [timer.py:197:stop] 0/618, RunningAvgSamplesPerSec=5.856869799235033, CurrSamplesPerSec=5.309910358634695, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 306/5000 [57:03<14:27:43, 11.09s/it][2022-12-20 11:19:34,395] [INFO] [logging.py:68:log_dist] [Rank 0] step=310, skipped=3, lr=[9.21513900295848e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:19:34,397] [INFO] [timer.py:197:stop] 0/620, RunningAvgSamplesPerSec=5.856720495963245, CurrSamplesPerSec=5.2251355620952395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 307/5000 [57:14<14:28:40, 11.11s/it][2022-12-20 11:19:45,455] [INFO] [timer.py:197:stop] 0/622, RunningAvgSamplesPerSec=5.856707967104583, CurrSamplesPerSec=5.296335751226638, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 308/5000 [57:25<14:26:49, 11.08s/it][2022-12-20 11:19:56,429] [INFO] [timer.py:197:stop] 0/624, RunningAvgSamplesPerSec=5.856858231009036, CurrSamplesPerSec=5.351022279040642, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 309/5000 [57:36<14:24:13, 11.05s/it][2022-12-20 11:20:07,459] [INFO] [timer.py:197:stop] 0/626, RunningAvgSamplesPerSec=5.8569274104854125, CurrSamplesPerSec=5.324807150009601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 310/5000 [57:48<14:24:10, 11.06s/it][2022-12-20 11:20:18,507] [INFO] [timer.py:197:stop] 0/628, RunningAvgSamplesPerSec=5.856929673395842, CurrSamplesPerSec=5.317838043138769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 311/5000 [57:59<14:23:24, 11.05s/it][2022-12-20 11:20:29,723] [INFO] [timer.py:197:stop] 0/630, RunningAvgSamplesPerSec=5.856646358612634, CurrSamplesPerSec=5.159234103306898, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▌         | 312/5000 [58:10<14:27:04, 11.10s/it][2022-12-20 11:20:40,716] [INFO] [timer.py:197:stop] 0/632, RunningAvgSamplesPerSec=5.856772581030724, CurrSamplesPerSec=5.3483946335670005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 313/5000 [58:21<14:23:04, 11.05s/it][2022-12-20 11:20:51,641] [INFO] [timer.py:197:stop] 0/634, RunningAvgSamplesPerSec=5.856990823872772, CurrSamplesPerSec=5.358690084323142, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 314/5000 [58:32<14:22:04, 11.04s/it][2022-12-20 11:21:02,862] [INFO] [timer.py:197:stop] 0/636, RunningAvgSamplesPerSec=5.856965989424361, CurrSamplesPerSec=5.310934018065791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 315/5000 [58:43<14:25:46, 11.09s/it][2022-12-20 11:21:13,895] [INFO] [timer.py:197:stop] 0/638, RunningAvgSamplesPerSec=5.8569925156153335, CurrSamplesPerSec=5.323889002135502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 316/5000 [58:54<14:24:34, 11.07s/it][2022-12-20 11:21:25,305] [INFO] [logging.py:68:log_dist] [Rank 0] step=320, skipped=3, lr=[9.266717518904195e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:21:25,312] [INFO] [timer.py:197:stop] 0/640, RunningAvgSamplesPerSec=5.856376790608875, CurrSamplesPerSec=5.013752677726196, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 317/5000 [59:05<14:33:06, 11.19s/it][2022-12-20 11:21:36,417] [INFO] [timer.py:197:stop] 0/642, RunningAvgSamplesPerSec=5.8563929235082135, CurrSamplesPerSec=5.297755015614669, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 318/5000 [59:16<14:29:55, 11.15s/it][2022-12-20 11:21:47,466] [INFO] [timer.py:197:stop] 0/644, RunningAvgSamplesPerSec=5.856395636667876, CurrSamplesPerSec=5.306099690579019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 319/5000 [59:27<14:26:57, 11.11s/it][2022-12-20 11:21:58,604] [INFO] [timer.py:197:stop] 0/646, RunningAvgSamplesPerSec=5.856250199114192, CurrSamplesPerSec=5.212158803054963, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 320/5000 [59:39<14:27:57, 11.13s/it][2022-12-20 11:22:09,588] [INFO] [timer.py:197:stop] 0/648, RunningAvgSamplesPerSec=5.85636027865457, CurrSamplesPerSec=5.368683994558905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 321/5000 [59:50<14:24:26, 11.08s/it][2022-12-20 11:22:20,631] [INFO] [timer.py:197:stop] 0/650, RunningAvgSamplesPerSec=5.856373236681452, CurrSamplesPerSec=5.317439221813549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 322/5000 [1:00:01<14:24:38, 11.09s/it][2022-12-20 11:22:31,695] [INFO] [timer.py:197:stop] 0/652, RunningAvgSamplesPerSec=5.856349350131584, CurrSamplesPerSec=5.349388199700322, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 323/5000 [1:00:12<14:21:30, 11.05s/it][2022-12-20 11:22:42,698] [INFO] [timer.py:197:stop] 0/654, RunningAvgSamplesPerSec=5.856433601225509, CurrSamplesPerSec=5.318656098442533, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 324/5000 [1:00:23<14:21:32, 11.05s/it][2022-12-20 11:22:54,145] [INFO] [timer.py:197:stop] 0/656, RunningAvgSamplesPerSec=5.855788467772298, CurrSamplesPerSec=4.997647766733765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  6%|▋         | 325/5000 [1:00:34<14:29:26, 11.16s/it]
                                                       
{'loss': 0.0094, 'learning_rate': 9.306822072655195e-06, 'epoch': 7.92}

  6%|▋         | 325/5000 [1:00:34<14:29:26, 11.16s/it][2022-12-20 11:23:05,138] [INFO] [timer.py:197:stop] 0/658, RunningAvgSamplesPerSec=5.855888650832801, CurrSamplesPerSec=5.336634028301657, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 326/5000 [1:00:45<14:25:56, 11.12s/it][2022-12-20 11:23:16,153] [INFO] [logging.py:68:log_dist] [Rank 0] step=330, skipped=3, lr=[9.316693955918556e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:23:16,154] [INFO] [timer.py:197:stop] 0/660, RunningAvgSamplesPerSec=5.855949758727352, CurrSamplesPerSec=5.335556744876842, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 327/5000 [1:00:56<14:23:49, 11.09s/it][2022-12-20 11:23:27,243] [INFO] [timer.py:197:stop] 0/662, RunningAvgSamplesPerSec=5.855892484447765, CurrSamplesPerSec=5.287509054943594, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 328/5000 [1:01:07<14:23:18, 11.09s/it][2022-12-20 11:23:37,391] [INFO] [timer.py:197:stop] 0/664, RunningAvgSamplesPerSec=5.85735323209044, CurrSamplesPerSec=6.233562566855649, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 11:23:48,448] [INFO] [timer.py:197:stop] 0/666, RunningAvgSamplesPerSec=5.857338227470666, CurrSamplesPerSec=5.267289773027849, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 329/5000 [1:01:24<16:23:35, 12.63s/it][2022-12-20 11:23:59,500] [INFO] [timer.py:197:stop] 0/668, RunningAvgSamplesPerSec=5.857347200215063, CurrSamplesPerSec=5.308694540566675, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 330/5000 [1:01:35<15:46:25, 12.16s/it][2022-12-20 11:24:10,543] [INFO] [timer.py:197:stop] 0/670, RunningAvgSamplesPerSec=5.857385076304811, CurrSamplesPerSec=5.301270596487957, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 331/5000 [1:01:46<15:20:08, 11.82s/it][2022-12-20 11:24:21,678] [INFO] [timer.py:197:stop] 0/672, RunningAvgSamplesPerSec=5.857249014526398, CurrSamplesPerSec=5.239804480385849, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 332/5000 [1:01:57<15:03:50, 11.62s/it][2022-12-20 11:24:32,746] [INFO] [timer.py:197:stop] 0/674, RunningAvgSamplesPerSec=5.857219671304712, CurrSamplesPerSec=5.305705983725652, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 333/5000 [1:02:08<14:50:52, 11.45s/it][2022-12-20 11:24:43,856] [INFO] [timer.py:197:stop] 0/676, RunningAvgSamplesPerSec=5.857177890691423, CurrSamplesPerSec=5.305336241949563, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 334/5000 [1:02:19<14:42:38, 11.35s/it][2022-12-20 11:24:55,100] [INFO] [timer.py:197:stop] 0/678, RunningAvgSamplesPerSec=5.856871282100234, CurrSamplesPerSec=5.381886777412229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 335/5000 [1:02:30<14:39:57, 11.32s/it][2022-12-20 11:25:06,146] [INFO] [logging.py:68:log_dist] [Rank 0] step=340, skipped=3, lr=[9.36516484736987e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:25:06,147] [INFO] [timer.py:197:stop] 0/680, RunningAvgSamplesPerSec=5.856874183841734, CurrSamplesPerSec=5.311983090019651, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 336/5000 [1:02:41<14:33:28, 11.24s/it][2022-12-20 11:25:17,264] [INFO] [timer.py:197:stop] 0/682, RunningAvgSamplesPerSec=5.856769826097349, CurrSamplesPerSec=5.260083967598557, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 337/5000 [1:02:52<14:30:30, 11.20s/it][2022-12-20 11:25:28,281] [INFO] [timer.py:197:stop] 0/684, RunningAvgSamplesPerSec=5.856848997686377, CurrSamplesPerSec=5.334556436166632, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 338/5000 [1:03:03<14:26:01, 11.15s/it][2022-12-20 11:25:39,356] [INFO] [timer.py:197:stop] 0/686, RunningAvgSamplesPerSec=5.856810932292181, CurrSamplesPerSec=5.311325979028653, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 339/5000 [1:03:14<14:24:10, 11.12s/it][2022-12-20 11:25:50,461] [INFO] [timer.py:197:stop] 0/688, RunningAvgSamplesPerSec=5.856725334978877, CurrSamplesPerSec=5.268608508754431, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 340/5000 [1:03:26<14:23:32, 11.12s/it][2022-12-20 11:26:01,513] [INFO] [timer.py:197:stop] 0/690, RunningAvgSamplesPerSec=5.85672288034997, CurrSamplesPerSec=5.321610103348116, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 341/5000 [1:03:37<14:21:48, 11.10s/it][2022-12-20 11:26:12,524] [INFO] [timer.py:197:stop] 0/692, RunningAvgSamplesPerSec=5.85678389857966, CurrSamplesPerSec=5.300367244818756, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 342/5000 [1:03:48<14:19:35, 11.07s/it][2022-12-20 11:26:23,575] [INFO] [timer.py:197:stop] 0/694, RunningAvgSamplesPerSec=5.856786210014206, CurrSamplesPerSec=5.34348785731348, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 343/5000 [1:03:59<14:18:53, 11.07s/it][2022-12-20 11:26:34,589] [INFO] [timer.py:197:stop] 0/696, RunningAvgSamplesPerSec=5.85684068935131, CurrSamplesPerSec=5.324141793871857, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 344/5000 [1:04:10<14:17:30, 11.05s/it][2022-12-20 11:26:45,606] [INFO] [timer.py:197:stop] 0/698, RunningAvgSamplesPerSec=5.856892485880058, CurrSamplesPerSec=5.30590901757704, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 345/5000 [1:04:21<14:16:32, 11.04s/it][2022-12-20 11:26:56,658] [INFO] [logging.py:68:log_dist] [Rank 0] step=350, skipped=3, lr=[9.412218256259678e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:26:56,659] [INFO] [timer.py:197:stop] 0/700, RunningAvgSamplesPerSec=5.856887313116275, CurrSamplesPerSec=5.323677410467434, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 346/5000 [1:04:32<14:16:39, 11.04s/it][2022-12-20 11:27:07,641] [INFO] [timer.py:197:stop] 0/702, RunningAvgSamplesPerSec=5.856992212460667, CurrSamplesPerSec=5.3356445575285605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 347/5000 [1:04:43<14:15:02, 11.03s/it][2022-12-20 11:27:18,698] [INFO] [timer.py:197:stop] 0/704, RunningAvgSamplesPerSec=5.856983178257457, CurrSamplesPerSec=5.328376718402791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 348/5000 [1:04:54<14:15:34, 11.03s/it][2022-12-20 11:27:29,726] [INFO] [timer.py:197:stop] 0/706, RunningAvgSamplesPerSec=5.857020604209413, CurrSamplesPerSec=5.3133792020873205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 349/5000 [1:05:05<14:15:13, 11.03s/it][2022-12-20 11:27:40,713] [INFO] [timer.py:197:stop] 0/708, RunningAvgSamplesPerSec=5.857116411543596, CurrSamplesPerSec=5.322566302405879, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 350/5000 [1:05:16<14:13:58, 11.02s/it]
                                                       
{'loss': 0.0086, 'learning_rate': 9.430661001703139e-06, 'epoch': 8.53}

  7%|▋         | 350/5000 [1:05:16<14:13:58, 11.02s/it][2022-12-20 11:27:51,695] [INFO] [timer.py:197:stop] 0/710, RunningAvgSamplesPerSec=5.857221730243372, CurrSamplesPerSec=5.309837255203636, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 351/5000 [1:05:27<14:12:55, 11.01s/it][2022-12-20 11:28:02,752] [INFO] [timer.py:197:stop] 0/712, RunningAvgSamplesPerSec=5.857208268604176, CurrSamplesPerSec=5.307402884686753, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 352/5000 [1:05:38<14:13:54, 11.02s/it][2022-12-20 11:28:13,799] [INFO] [timer.py:197:stop] 0/714, RunningAvgSamplesPerSec=5.857214433634748, CurrSamplesPerSec=5.31830899478296, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 353/5000 [1:05:49<14:14:16, 11.03s/it][2022-12-20 11:28:24,807] [INFO] [timer.py:197:stop] 0/716, RunningAvgSamplesPerSec=5.857289469021173, CurrSamplesPerSec=5.332237045189865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 354/5000 [1:06:00<14:13:35, 11.02s/it][2022-12-20 11:28:35,790] [INFO] [timer.py:197:stop] 0/718, RunningAvgSamplesPerSec=5.857388606379537, CurrSamplesPerSec=5.344134012859187, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 355/5000 [1:06:11<14:12:28, 11.01s/it][2022-12-20 11:28:46,833] [INFO] [logging.py:68:log_dist] [Rank 0] step=360, skipped=3, lr=[9.457934738108298e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:28:46,835] [INFO] [timer.py:197:stop] 0/720, RunningAvgSamplesPerSec=5.857395722103578, CurrSamplesPerSec=5.311728298321715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 356/5000 [1:06:22<14:13:02, 11.02s/it][2022-12-20 11:28:57,822] [INFO] [timer.py:197:stop] 0/722, RunningAvgSamplesPerSec=5.857489327340425, CurrSamplesPerSec=5.333657814268813, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 357/5000 [1:06:33<14:12:04, 11.01s/it][2022-12-20 11:29:08,872] [INFO] [timer.py:197:stop] 0/724, RunningAvgSamplesPerSec=5.8574881571178326, CurrSamplesPerSec=5.314068801766477, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 358/5000 [1:06:44<14:12:47, 11.02s/it][2022-12-20 11:29:19,910] [INFO] [timer.py:197:stop] 0/726, RunningAvgSamplesPerSec=5.857506353556551, CurrSamplesPerSec=5.292718803918366, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 359/5000 [1:06:55<14:12:56, 11.03s/it][2022-12-20 11:29:30,937] [INFO] [timer.py:197:stop] 0/728, RunningAvgSamplesPerSec=5.85756335363611, CurrSamplesPerSec=5.325089818273374, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 360/5000 [1:07:06<14:12:46, 11.03s/it][2022-12-20 11:29:42,184] [INFO] [timer.py:197:stop] 0/730, RunningAvgSamplesPerSec=5.857270387894122, CurrSamplesPerSec=5.135831221683477, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 361/5000 [1:07:17<14:17:41, 11.09s/it][2022-12-20 11:29:53,261] [INFO] [timer.py:197:stop] 0/732, RunningAvgSamplesPerSec=5.857231214736918, CurrSamplesPerSec=5.289228309333904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 362/5000 [1:07:28<14:17:09, 11.09s/it][2022-12-20 11:30:04,310] [INFO] [timer.py:197:stop] 0/734, RunningAvgSamplesPerSec=5.85725899352918, CurrSamplesPerSec=5.32176413559539, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 363/5000 [1:07:39<14:16:02, 11.08s/it][2022-12-20 11:30:15,473] [INFO] [timer.py:197:stop] 0/736, RunningAvgSamplesPerSec=5.857104886636695, CurrSamplesPerSec=5.312392447662331, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 364/5000 [1:07:51<14:17:50, 11.10s/it][2022-12-20 11:30:26,523] [INFO] [timer.py:197:stop] 0/738, RunningAvgSamplesPerSec=5.857103879464872, CurrSamplesPerSec=5.301156692316084, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 365/5000 [1:08:02<14:16:26, 11.09s/it][2022-12-20 11:30:37,554] [INFO] [logging.py:68:log_dist] [Rank 0] step=370, skipped=3, lr=[9.502388170790473e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:30:37,556] [INFO] [timer.py:197:stop] 0/740, RunningAvgSamplesPerSec=5.857129998178556, CurrSamplesPerSec=5.287700074305617, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 366/5000 [1:08:13<14:15:00, 11.07s/it][2022-12-20 11:30:48,600] [INFO] [timer.py:197:stop] 0/742, RunningAvgSamplesPerSec=5.857163079603807, CurrSamplesPerSec=5.325600091451728, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 367/5000 [1:08:24<14:14:13, 11.06s/it][2022-12-20 11:30:59,617] [INFO] [timer.py:197:stop] 0/744, RunningAvgSamplesPerSec=5.857210600477119, CurrSamplesPerSec=5.332680887998972, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 368/5000 [1:08:35<14:12:58, 11.05s/it][2022-12-20 11:31:10,723] [INFO] [timer.py:197:stop] 0/746, RunningAvgSamplesPerSec=5.857154843293825, CurrSamplesPerSec=5.298692619619755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 369/5000 [1:08:46<14:14:07, 11.07s/it][2022-12-20 11:31:20,838] [INFO] [timer.py:197:stop] 0/748, RunningAvgSamplesPerSec=5.858487066499258, CurrSamplesPerSec=5.3364848628851655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 370/5000 [1:09:01<15:46:55, 12.27s/it][2022-12-20 11:31:31,943] [INFO] [timer.py:197:stop] 0/750, RunningAvgSamplesPerSec=5.858402983731458, CurrSamplesPerSec=5.268743562392782, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 371/5000 [1:09:12<15:27:36, 12.02s/it][2022-12-20 11:31:43,309] [INFO] [timer.py:197:stop] 0/752, RunningAvgSamplesPerSec=5.858393415299045, CurrSamplesPerSec=5.338630422021463, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 372/5000 [1:09:23<15:02:22, 11.70s/it][2022-12-20 11:31:54,259] [INFO] [timer.py:197:stop] 0/754, RunningAvgSamplesPerSec=5.858531485306879, CurrSamplesPerSec=5.331005477142798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 373/5000 [1:09:34<14:46:37, 11.50s/it][2022-12-20 11:32:05,392] [INFO] [timer.py:197:stop] 0/756, RunningAvgSamplesPerSec=5.858409888978332, CurrSamplesPerSec=5.2387555032356365, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  7%|▋         | 374/5000 [1:09:45<14:38:10, 11.39s/it][2022-12-20 11:32:16,413] [INFO] [timer.py:197:stop] 0/758, RunningAvgSamplesPerSec=5.85844702946053, CurrSamplesPerSec=5.342470750253694, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 375/5000 [1:09:56<14:29:31, 11.28s/it]
                                                       
{'loss': 0.0094, 'learning_rate': 9.541372600623587e-06, 'epoch': 9.14}

  8%|▊         | 375/5000 [1:09:56<14:29:31, 11.28s/it][2022-12-20 11:32:27,456] [INFO] [logging.py:68:log_dist] [Rank 0] step=380, skipped=3, lr=[9.545646472790669e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:32:27,457] [INFO] [timer.py:197:stop] 0/760, RunningAvgSamplesPerSec=5.858475773923775, CurrSamplesPerSec=5.325791548355074, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 376/5000 [1:10:08<14:23:36, 11.21s/it][2022-12-20 11:32:38,501] [INFO] [timer.py:197:stop] 0/762, RunningAvgSamplesPerSec=5.858481340277548, CurrSamplesPerSec=5.315939279238003, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 377/5000 [1:10:19<14:18:28, 11.14s/it][2022-12-20 11:32:49,488] [INFO] [timer.py:197:stop] 0/764, RunningAvgSamplesPerSec=5.858565661727381, CurrSamplesPerSec=5.3201649556522534, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 378/5000 [1:10:30<14:15:56, 11.11s/it][2022-12-20 11:33:00,550] [INFO] [timer.py:197:stop] 0/766, RunningAvgSamplesPerSec=5.858543449701859, CurrSamplesPerSec=5.300038011495693, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 379/5000 [1:10:41<14:13:35, 11.08s/it][2022-12-20 11:33:11,518] [INFO] [timer.py:197:stop] 0/768, RunningAvgSamplesPerSec=5.858654259250757, CurrSamplesPerSec=5.34410039274746, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 380/5000 [1:10:52<14:10:45, 11.05s/it][2022-12-20 11:33:22,516] [INFO] [timer.py:197:stop] 0/770, RunningAvgSamplesPerSec=5.858723962983873, CurrSamplesPerSec=5.319175679470606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 381/5000 [1:11:03<14:10:36, 11.05s/it][2022-12-20 11:33:33,576] [INFO] [timer.py:197:stop] 0/772, RunningAvgSamplesPerSec=5.85872978132438, CurrSamplesPerSec=5.323809178153291, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 382/5000 [1:11:14<14:10:34, 11.05s/it][2022-12-20 11:33:44,680] [INFO] [timer.py:197:stop] 0/774, RunningAvgSamplesPerSec=5.8586514844308555, CurrSamplesPerSec=5.267953193628031, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 383/5000 [1:11:25<14:12:03, 11.07s/it][2022-12-20 11:33:55,783] [INFO] [timer.py:197:stop] 0/776, RunningAvgSamplesPerSec=5.858573812970191, CurrSamplesPerSec=5.285673722936061, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 384/5000 [1:11:36<14:11:53, 11.07s/it][2022-12-20 11:34:06,787] [INFO] [timer.py:197:stop] 0/778, RunningAvgSamplesPerSec=5.858655557151274, CurrSamplesPerSec=5.346452692872004, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 385/5000 [1:11:47<14:09:41, 11.05s/it][2022-12-20 11:34:17,809] [INFO] [logging.py:68:log_dist] [Rank 0] step=390, skipped=3, lr=[9.587772227411331e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:34:17,810] [INFO] [timer.py:197:stop] 0/780, RunningAvgSamplesPerSec=5.858687000371175, CurrSamplesPerSec=5.313428423221936, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 386/5000 [1:11:58<14:09:14, 11.04s/it][2022-12-20 11:34:28,847] [INFO] [timer.py:197:stop] 0/782, RunningAvgSamplesPerSec=5.858701994505354, CurrSamplesPerSec=5.313331033569245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 387/5000 [1:12:09<14:09:13, 11.05s/it][2022-12-20 11:34:39,923] [INFO] [timer.py:197:stop] 0/784, RunningAvgSamplesPerSec=5.85866232163694, CurrSamplesPerSec=5.290658573580088, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 388/5000 [1:12:20<14:09:09, 11.05s/it][2022-12-20 11:34:51,001] [INFO] [timer.py:197:stop] 0/786, RunningAvgSamplesPerSec=5.8586200109629925, CurrSamplesPerSec=5.266420281499677, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 389/5000 [1:12:31<14:10:55, 11.07s/it][2022-12-20 11:35:02,084] [INFO] [timer.py:197:stop] 0/788, RunningAvgSamplesPerSec=5.85857305580418, CurrSamplesPerSec=5.308749974290375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 390/5000 [1:12:42<14:10:15, 11.07s/it][2022-12-20 11:35:13,157] [INFO] [timer.py:197:stop] 0/790, RunningAvgSamplesPerSec=5.858537954315953, CurrSamplesPerSec=5.290325123612498, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 391/5000 [1:12:53<14:09:44, 11.06s/it][2022-12-20 11:35:24,176] [INFO] [timer.py:197:stop] 0/792, RunningAvgSamplesPerSec=5.858576169537123, CurrSamplesPerSec=5.3190329690118485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 392/5000 [1:13:04<14:09:31, 11.06s/it][2022-12-20 11:35:35,284] [INFO] [timer.py:197:stop] 0/794, RunningAvgSamplesPerSec=5.858493610773851, CurrSamplesPerSec=5.277612669913634, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 393/5000 [1:13:15<14:09:46, 11.07s/it][2022-12-20 11:35:46,309] [INFO] [timer.py:197:stop] 0/796, RunningAvgSamplesPerSec=5.858543006036668, CurrSamplesPerSec=5.3259020755497275, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 394/5000 [1:13:26<14:09:58, 11.07s/it][2022-12-20 11:35:57,429] [INFO] [timer.py:197:stop] 0/798, RunningAvgSamplesPerSec=5.858444362187369, CurrSamplesPerSec=5.2940112564528015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 395/5000 [1:13:37<14:08:33, 11.06s/it][2022-12-20 11:36:08,405] [INFO] [logging.py:68:log_dist] [Rank 0] step=400, skipped=3, lr=[9.62882322733502e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:36:08,406] [INFO] [timer.py:197:stop] 0/800, RunningAvgSamplesPerSec=5.858539326301925, CurrSamplesPerSec=5.330597480255167, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 396/5000 [1:13:48<14:07:09, 11.04s/it][2022-12-20 11:36:19,459] [INFO] [timer.py:197:stop] 0/802, RunningAvgSamplesPerSec=5.85855019943057, CurrSamplesPerSec=5.288017151084738, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 397/5000 [1:13:59<14:07:16, 11.04s/it][2022-12-20 11:36:30,505] [INFO] [timer.py:197:stop] 0/804, RunningAvgSamplesPerSec=5.858574508655286, CurrSamplesPerSec=5.294008541870786, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 398/5000 [1:14:11<14:06:41, 11.04s/it][2022-12-20 11:36:41,486] [INFO] [timer.py:197:stop] 0/806, RunningAvgSamplesPerSec=5.85866191152302, CurrSamplesPerSec=5.333252378209095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 399/5000 [1:14:22<14:06:18, 11.04s/it][2022-12-20 11:36:52,492] [INFO] [timer.py:197:stop] 0/808, RunningAvgSamplesPerSec=5.858717459007704, CurrSamplesPerSec=5.356697262059558, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 400/5000 [1:14:33<14:05:33, 11.03s/it]
                                                       
{'loss': 0.0076, 'learning_rate': 9.644954810309533e-06, 'epoch': 9.75}

  8%|▊         | 400/5000 [1:14:33<14:05:33, 11.03s/it][2022-12-20 11:37:03,573] [INFO] [timer.py:197:stop] 0/810, RunningAvgSamplesPerSec=5.8586731509075305, CurrSamplesPerSec=5.295601853641954, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 401/5000 [1:14:44<14:05:48, 11.03s/it][2022-12-20 11:37:14,606] [INFO] [timer.py:197:stop] 0/812, RunningAvgSamplesPerSec=5.858691521744896, CurrSamplesPerSec=5.308055666476981, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 402/5000 [1:14:55<14:05:58, 11.04s/it][2022-12-20 11:37:25,609] [INFO] [timer.py:197:stop] 0/814, RunningAvgSamplesPerSec=5.858748279894398, CurrSamplesPerSec=5.349244929460072, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 403/5000 [1:15:06<14:05:28, 11.04s/it][2022-12-20 11:37:36,691] [INFO] [timer.py:197:stop] 0/816, RunningAvgSamplesPerSec=5.858725528545898, CurrSamplesPerSec=5.31423418082853, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 404/5000 [1:15:17<14:05:03, 11.03s/it][2022-12-20 11:37:47,675] [INFO] [timer.py:197:stop] 0/818, RunningAvgSamplesPerSec=5.858808372787004, CurrSamplesPerSec=5.335353768722849, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 405/5000 [1:15:28<14:04:44, 11.03s/it][2022-12-20 11:37:58,735] [INFO] [logging.py:68:log_dist] [Rank 0] step=410, skipped=3, lr=[9.668852951432537e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:37:58,737] [INFO] [timer.py:197:stop] 0/820, RunningAvgSamplesPerSec=5.858787384887806, CurrSamplesPerSec=5.30357548518046, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 406/5000 [1:15:39<14:04:06, 11.02s/it][2022-12-20 11:38:09,719] [INFO] [timer.py:197:stop] 0/822, RunningAvgSamplesPerSec=5.858871106206928, CurrSamplesPerSec=5.329114650802879, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 407/5000 [1:15:50<14:03:53, 11.02s/it][2022-12-20 11:38:20,777] [INFO] [timer.py:197:stop] 0/824, RunningAvgSamplesPerSec=5.858856309268045, CurrSamplesPerSec=5.29874951816173, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 408/5000 [1:16:01<14:04:25, 11.03s/it][2022-12-20 11:38:31,829] [INFO] [timer.py:197:stop] 0/826, RunningAvgSamplesPerSec=5.858847677827434, CurrSamplesPerSec=5.3011541797792345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 409/5000 [1:16:12<14:04:54, 11.04s/it][2022-12-20 11:38:42,840] [INFO] [timer.py:197:stop] 0/828, RunningAvgSamplesPerSec=5.858893908404291, CurrSamplesPerSec=5.34684757127229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 410/5000 [1:16:23<14:04:08, 11.03s/it][2022-12-20 11:38:52,975] [INFO] [timer.py:197:stop] 0/830, RunningAvgSamplesPerSec=5.860060641552665, CurrSamplesPerSec=6.257422271354992, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 11:39:04,007] [INFO] [timer.py:197:stop] 0/832, RunningAvgSamplesPerSec=5.860072465867985, CurrSamplesPerSec=5.318516787888, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 411/5000 [1:16:39<16:02:27, 12.58s/it][2022-12-20 11:39:15,023] [INFO] [timer.py:197:stop] 0/834, RunningAvgSamplesPerSec=5.860109251845212, CurrSamplesPerSec=5.34325236960056, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 412/5000 [1:16:50<15:26:17, 12.11s/it][2022-12-20 11:39:26,112] [INFO] [timer.py:197:stop] 0/836, RunningAvgSamplesPerSec=5.860073444704911, CurrSamplesPerSec=5.302166087258334, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 413/5000 [1:17:01<15:02:34, 11.81s/it][2022-12-20 11:39:37,137] [INFO] [timer.py:197:stop] 0/838, RunningAvgSamplesPerSec=5.860097299790324, CurrSamplesPerSec=5.291036075487514, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 414/5000 [1:17:12<14:44:28, 11.57s/it][2022-12-20 11:39:48,242] [INFO] [logging.py:68:log_dist] [Rank 0] step=420, skipped=3, lr=[9.707910983687811e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:39:48,243] [INFO] [timer.py:197:stop] 0/840, RunningAvgSamplesPerSec=5.860017899776563, CurrSamplesPerSec=5.226937636032848, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 415/5000 [1:17:23<14:33:34, 11.43s/it][2022-12-20 11:39:59,243] [INFO] [timer.py:197:stop] 0/842, RunningAvgSamplesPerSec=5.860093437369332, CurrSamplesPerSec=5.3351349026522135, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 416/5000 [1:17:34<14:23:31, 11.30s/it][2022-12-20 11:40:10,292] [INFO] [timer.py:197:stop] 0/844, RunningAvgSamplesPerSec=5.860089215074601, CurrSamplesPerSec=5.323641513350243, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 417/5000 [1:17:45<14:17:30, 11.23s/it][2022-12-20 11:40:21,320] [INFO] [timer.py:197:stop] 0/846, RunningAvgSamplesPerSec=5.860109413931928, CurrSamplesPerSec=5.324137781117111, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 418/5000 [1:17:56<14:12:46, 11.17s/it][2022-12-20 11:40:32,345] [INFO] [timer.py:197:stop] 0/848, RunningAvgSamplesPerSec=5.860133538018046, CurrSamplesPerSec=5.307025981578195, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 419/5000 [1:18:07<14:09:20, 11.12s/it][2022-12-20 11:40:43,398] [INFO] [timer.py:197:stop] 0/850, RunningAvgSamplesPerSec=5.8601241968027535, CurrSamplesPerSec=5.323417906326408, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 420/5000 [1:18:18<14:07:31, 11.10s/it][2022-12-20 11:40:54,451] [INFO] [timer.py:197:stop] 0/852, RunningAvgSamplesPerSec=5.860134582198357, CurrSamplesPerSec=5.305974251792388, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 421/5000 [1:18:30<14:06:11, 11.09s/it][2022-12-20 11:41:05,415] [INFO] [timer.py:197:stop] 0/854, RunningAvgSamplesPerSec=5.860236668377545, CurrSamplesPerSec=5.3664307481615605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 422/5000 [1:18:41<14:03:09, 11.05s/it][2022-12-20 11:41:16,463] [INFO] [timer.py:197:stop] 0/856, RunningAvgSamplesPerSec=5.860234674838121, CurrSamplesPerSec=5.328476352729191, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 423/5000 [1:18:52<14:02:55, 11.05s/it][2022-12-20 11:41:27,459] [INFO] [timer.py:197:stop] 0/858, RunningAvgSamplesPerSec=5.860294895253216, CurrSamplesPerSec=5.335628649225097, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 424/5000 [1:19:03<14:01:29, 11.03s/it][2022-12-20 11:41:38,479] [INFO] [logging.py:68:log_dist] [Rank 0] step=430, skipped=3, lr=[9.746043382472281e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:41:38,480] [INFO] [timer.py:197:stop] 0/860, RunningAvgSamplesPerSec=5.860323407398255, CurrSamplesPerSec=5.316136569590542, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  8%|▊         | 425/5000 [1:19:14<14:01:01, 11.03s/it]
                                                       
{'loss': 0.0072, 'learning_rate': 9.746043382472281e-06, 'epoch': 10.36}

  8%|▊         | 425/5000 [1:19:14<14:01:01, 11.03s/it][2022-12-20 11:41:49,510] [INFO] [timer.py:197:stop] 0/862, RunningAvgSamplesPerSec=5.8603625017808, CurrSamplesPerSec=5.328156097688009, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 426/5000 [1:19:25<14:00:51, 11.03s/it][2022-12-20 11:42:00,463] [INFO] [timer.py:197:stop] 0/864, RunningAvgSamplesPerSec=5.860476284261011, CurrSamplesPerSec=5.358477428858241, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 427/5000 [1:19:36<13:58:54, 11.01s/it][2022-12-20 11:42:11,487] [INFO] [timer.py:197:stop] 0/866, RunningAvgSamplesPerSec=5.860500265437958, CurrSamplesPerSec=5.306487161875229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 428/5000 [1:19:47<13:59:06, 11.01s/it][2022-12-20 11:42:22,567] [INFO] [timer.py:197:stop] 0/868, RunningAvgSamplesPerSec=5.860455751218728, CurrSamplesPerSec=5.2846972311765406, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 429/5000 [1:19:58<14:00:28, 11.03s/it][2022-12-20 11:42:33,559] [INFO] [timer.py:197:stop] 0/870, RunningAvgSamplesPerSec=5.8605199572343105, CurrSamplesPerSec=5.347311532065747, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 430/5000 [1:20:09<13:59:22, 11.02s/it][2022-12-20 11:42:44,634] [INFO] [timer.py:197:stop] 0/872, RunningAvgSamplesPerSec=5.860496447635159, CurrSamplesPerSec=5.30912334217622, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 431/5000 [1:20:20<14:00:26, 11.04s/it][2022-12-20 11:42:55,692] [INFO] [timer.py:197:stop] 0/874, RunningAvgSamplesPerSec=5.86048003466388, CurrSamplesPerSec=5.298556235442738, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 432/5000 [1:20:31<14:00:44, 11.04s/it][2022-12-20 11:43:06,736] [INFO] [timer.py:197:stop] 0/876, RunningAvgSamplesPerSec=5.860479558052339, CurrSamplesPerSec=5.299038004205269, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 433/5000 [1:20:42<14:00:36, 11.04s/it][2022-12-20 11:43:17,820] [INFO] [timer.py:197:stop] 0/878, RunningAvgSamplesPerSec=5.860452781797949, CurrSamplesPerSec=5.3075586141825575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 434/5000 [1:20:53<14:01:19, 11.06s/it][2022-12-20 11:43:28,839] [INFO] [logging.py:68:log_dist] [Rank 0] step=440, skipped=3, lr=[9.783293007067022e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:43:28,841] [INFO] [timer.py:197:stop] 0/880, RunningAvgSamplesPerSec=5.860481629585393, CurrSamplesPerSec=5.317008233381236, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 435/5000 [1:21:04<14:00:21, 11.05s/it][2022-12-20 11:43:39,898] [INFO] [timer.py:197:stop] 0/882, RunningAvgSamplesPerSec=5.860468482483648, CurrSamplesPerSec=5.323748994987532, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 436/5000 [1:21:15<14:00:26, 11.05s/it][2022-12-20 11:43:50,913] [INFO] [timer.py:197:stop] 0/884, RunningAvgSamplesPerSec=5.860517553136644, CurrSamplesPerSec=5.316391994967286, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▊         | 437/5000 [1:21:26<13:59:29, 11.04s/it][2022-12-20 11:44:01,912] [INFO] [timer.py:197:stop] 0/886, RunningAvgSamplesPerSec=5.86057302672167, CurrSamplesPerSec=5.320741148736618, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 438/5000 [1:21:37<13:58:23, 11.03s/it][2022-12-20 11:44:12,926] [INFO] [timer.py:197:stop] 0/888, RunningAvgSamplesPerSec=5.860609647029968, CurrSamplesPerSec=5.340411560837895, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 439/5000 [1:21:48<13:57:55, 11.02s/it][2022-12-20 11:44:23,918] [INFO] [timer.py:197:stop] 0/890, RunningAvgSamplesPerSec=5.8606731771648, CurrSamplesPerSec=5.341738043609252, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 440/5000 [1:21:59<13:57:01, 11.01s/it][2022-12-20 11:44:34,980] [INFO] [timer.py:197:stop] 0/892, RunningAvgSamplesPerSec=5.860652373472958, CurrSamplesPerSec=5.312009369431809, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 441/5000 [1:22:10<13:57:57, 11.03s/it][2022-12-20 11:44:46,104] [INFO] [timer.py:197:stop] 0/894, RunningAvgSamplesPerSec=5.8605554535418785, CurrSamplesPerSec=5.27261093490922, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 442/5000 [1:22:21<13:59:57, 11.06s/it][2022-12-20 11:44:57,168] [INFO] [timer.py:197:stop] 0/896, RunningAvgSamplesPerSec=5.860533492349253, CurrSamplesPerSec=5.30707487522904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 443/5000 [1:22:32<13:59:56, 11.06s/it][2022-12-20 11:45:08,199] [INFO] [timer.py:197:stop] 0/898, RunningAvgSamplesPerSec=5.860558584403761, CurrSamplesPerSec=5.312049735115313, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 444/5000 [1:22:43<13:59:06, 11.05s/it][2022-12-20 11:45:19,228] [INFO] [logging.py:68:log_dist] [Rank 0] step=450, skipped=3, lr=[9.819699807237934e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:45:19,229] [INFO] [timer.py:197:stop] 0/900, RunningAvgSamplesPerSec=5.860593790128849, CurrSamplesPerSec=5.343844425641268, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 445/5000 [1:22:54<13:58:26, 11.04s/it][2022-12-20 11:45:30,282] [INFO] [timer.py:197:stop] 0/902, RunningAvgSamplesPerSec=5.860583711612014, CurrSamplesPerSec=5.31832964692065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 446/5000 [1:23:05<13:58:27, 11.05s/it][2022-12-20 11:45:41,279] [INFO] [timer.py:197:stop] 0/904, RunningAvgSamplesPerSec=5.860640328913929, CurrSamplesPerSec=5.3695918809052205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 447/5000 [1:23:16<13:57:08, 11.03s/it][2022-12-20 11:45:52,339] [INFO] [timer.py:197:stop] 0/906, RunningAvgSamplesPerSec=5.860621878084019, CurrSamplesPerSec=5.298790728530523, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 448/5000 [1:23:27<13:57:35, 11.04s/it][2022-12-20 11:46:03,319] [INFO] [timer.py:197:stop] 0/908, RunningAvgSamplesPerSec=5.860698608716086, CurrSamplesPerSec=5.3298631585497525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 449/5000 [1:23:38<13:56:02, 11.02s/it][2022-12-20 11:46:14,327] [INFO] [timer.py:197:stop] 0/910, RunningAvgSamplesPerSec=5.8607422426672855, CurrSamplesPerSec=5.328692768783543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 450/5000 [1:23:49<13:55:32, 11.02s/it]
                                                       
{'loss': 0.0065, 'learning_rate': 9.837598900861371e-06, 'epoch': 10.96}

  9%|▉         | 450/5000 [1:23:49<13:55:32, 11.02s/it][2022-12-20 11:46:25,265] [INFO] [timer.py:197:stop] 0/912, RunningAvgSamplesPerSec=5.860868710563395, CurrSamplesPerSec=5.3515531110366386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 451/5000 [1:24:00<13:53:31, 10.99s/it][2022-12-20 11:46:35,339] [INFO] [timer.py:197:stop] 0/914, RunningAvgSamplesPerSec=5.861998185578178, CurrSamplesPerSec=5.354159299187644, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 452/5000 [1:24:15<15:24:59, 12.20s/it][2022-12-20 11:46:46,341] [INFO] [timer.py:197:stop] 0/916, RunningAvgSamplesPerSec=5.862041748616294, CurrSamplesPerSec=5.343533808555723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 453/5000 [1:24:26<14:57:35, 11.84s/it][2022-12-20 11:46:57,389] [INFO] [timer.py:197:stop] 0/918, RunningAvgSamplesPerSec=5.862032959351219, CurrSamplesPerSec=5.308216262811697, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 454/5000 [1:24:37<14:39:15, 11.60s/it][2022-12-20 11:47:08,430] [INFO] [logging.py:68:log_dist] [Rank 0] step=460, skipped=3, lr=[9.855301080769973e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:47:08,432] [INFO] [timer.py:197:stop] 0/920, RunningAvgSamplesPerSec=5.862028959689885, CurrSamplesPerSec=5.310031782000001, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 455/5000 [1:24:48<14:26:40, 11.44s/it][2022-12-20 11:47:19,523] [INFO] [timer.py:197:stop] 0/922, RunningAvgSamplesPerSec=5.861981053014086, CurrSamplesPerSec=5.290328459989351, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 456/5000 [1:25:00<14:17:10, 11.32s/it][2022-12-20 11:47:30,554] [INFO] [timer.py:197:stop] 0/924, RunningAvgSamplesPerSec=5.861993517492389, CurrSamplesPerSec=5.283585069600267, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 457/5000 [1:25:11<14:11:36, 11.25s/it][2022-12-20 11:47:41,573] [INFO] [timer.py:197:stop] 0/926, RunningAvgSamplesPerSec=5.8620255542311135, CurrSamplesPerSec=5.337807272625028, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 458/5000 [1:25:22<14:04:29, 11.16s/it][2022-12-20 11:47:52,542] [INFO] [timer.py:197:stop] 0/928, RunningAvgSamplesPerSec=5.8621094423400715, CurrSamplesPerSec=5.314820033749798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 459/5000 [1:25:33<14:01:10, 11.11s/it][2022-12-20 11:48:03,579] [INFO] [timer.py:197:stop] 0/930, RunningAvgSamplesPerSec=5.862118109315523, CurrSamplesPerSec=5.30061069029084, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 460/5000 [1:25:44<13:59:38, 11.10s/it][2022-12-20 11:48:14,660] [INFO] [timer.py:197:stop] 0/932, RunningAvgSamplesPerSec=5.862071170733892, CurrSamplesPerSec=5.275031937828873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 461/5000 [1:25:55<13:59:18, 11.09s/it][2022-12-20 11:48:25,697] [INFO] [timer.py:197:stop] 0/934, RunningAvgSamplesPerSec=5.862079144101076, CurrSamplesPerSec=5.321313247170106, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 462/5000 [1:26:06<13:58:24, 11.09s/it][2022-12-20 11:48:36,815] [INFO] [timer.py:197:stop] 0/936, RunningAvgSamplesPerSec=5.861989910029147, CurrSamplesPerSec=5.2734932462821575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 463/5000 [1:26:17<13:58:37, 11.09s/it][2022-12-20 11:48:47,841] [INFO] [timer.py:197:stop] 0/938, RunningAvgSamplesPerSec=5.862009920510047, CurrSamplesPerSec=5.343665071781384, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 464/5000 [1:26:28<13:56:59, 11.07s/it][2022-12-20 11:48:58,972] [INFO] [logging.py:68:log_dist] [Rank 0] step=470, skipped=3, lr=[9.890131703122795e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:48:58,974] [INFO] [timer.py:197:stop] 0/940, RunningAvgSamplesPerSec=5.861906613850916, CurrSamplesPerSec=5.250268727791255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 465/5000 [1:26:39<13:57:03, 11.07s/it][2022-12-20 11:49:09,980] [INFO] [timer.py:197:stop] 0/942, RunningAvgSamplesPerSec=5.861954180731467, CurrSamplesPerSec=5.320383227879851, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 466/5000 [1:26:50<13:56:31, 11.07s/it][2022-12-20 11:49:21,066] [INFO] [timer.py:197:stop] 0/944, RunningAvgSamplesPerSec=5.861903964263766, CurrSamplesPerSec=5.291632471413188, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 467/5000 [1:27:01<13:56:29, 11.07s/it][2022-12-20 11:49:32,140] [INFO] [timer.py:197:stop] 0/946, RunningAvgSamplesPerSec=5.861867430291547, CurrSamplesPerSec=5.294051349065213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 468/5000 [1:27:12<13:55:26, 11.06s/it][2022-12-20 11:49:43,147] [INFO] [timer.py:197:stop] 0/948, RunningAvgSamplesPerSec=5.861906852801518, CurrSamplesPerSec=5.317812548807862, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 469/5000 [1:27:23<13:54:45, 11.05s/it][2022-12-20 11:49:54,186] [INFO] [timer.py:197:stop] 0/950, RunningAvgSamplesPerSec=5.861909198031919, CurrSamplesPerSec=5.317228774451515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 470/5000 [1:27:34<13:53:58, 11.05s/it][2022-12-20 11:50:05,209] [INFO] [timer.py:197:stop] 0/952, RunningAvgSamplesPerSec=5.861936751162797, CurrSamplesPerSec=5.324531059834072, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 471/5000 [1:27:45<13:53:54, 11.05s/it][2022-12-20 11:50:16,288] [INFO] [timer.py:197:stop] 0/954, RunningAvgSamplesPerSec=5.8618996188265, CurrSamplesPerSec=5.2972937601188335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 472/5000 [1:27:56<13:53:35, 11.05s/it][2022-12-20 11:50:27,329] [INFO] [timer.py:197:stop] 0/956, RunningAvgSamplesPerSec=5.861901575343901, CurrSamplesPerSec=5.297558250839959, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 473/5000 [1:28:07<13:53:12, 11.04s/it][2022-12-20 11:50:38,356] [INFO] [timer.py:197:stop] 0/958, RunningAvgSamplesPerSec=5.86191686038331, CurrSamplesPerSec=5.3055176459673525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

  9%|▉         | 474/5000 [1:28:18<13:53:26, 11.05s/it][2022-12-20 11:50:49,352] [INFO] [logging.py:68:log_dist] [Rank 0] step=480, skipped=3, lr=[9.924224332752688e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:50:49,353] [INFO] [timer.py:197:stop] 0/960, RunningAvgSamplesPerSec=5.861965862981737, CurrSamplesPerSec=5.362801672189982, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 475/5000 [1:28:29<13:51:28, 11.02s/it]
                                                       
{'loss': 0.0052, 'learning_rate': 9.924224332752688e-06, 'epoch': 11.58}

 10%|▉         | 475/5000 [1:28:29<13:51:28, 11.02s/it][2022-12-20 11:51:00,393] [INFO] [timer.py:197:stop] 0/962, RunningAvgSamplesPerSec=5.861974887261498, CurrSamplesPerSec=5.306702634663121, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 476/5000 [1:28:40<13:52:00, 11.03s/it][2022-12-20 11:51:11,392] [INFO] [timer.py:197:stop] 0/964, RunningAvgSamplesPerSec=5.862021898265052, CurrSamplesPerSec=5.352004230316168, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 477/5000 [1:28:51<13:51:04, 11.02s/it][2022-12-20 11:51:22,464] [INFO] [timer.py:197:stop] 0/966, RunningAvgSamplesPerSec=5.861987983815448, CurrSamplesPerSec=5.28981178876307, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 478/5000 [1:29:03<13:52:32, 11.05s/it][2022-12-20 11:51:33,574] [INFO] [timer.py:197:stop] 0/968, RunningAvgSamplesPerSec=5.861932537300607, CurrSamplesPerSec=5.294747219668232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 479/5000 [1:29:14<13:52:16, 11.05s/it][2022-12-20 11:51:44,597] [INFO] [timer.py:197:stop] 0/970, RunningAvgSamplesPerSec=5.861951543704444, CurrSamplesPerSec=5.296084966077177, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 480/5000 [1:29:25<13:52:48, 11.06s/it][2022-12-20 11:51:55,697] [INFO] [timer.py:197:stop] 0/972, RunningAvgSamplesPerSec=5.861894334524614, CurrSamplesPerSec=5.277196827636214, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 481/5000 [1:29:36<13:53:11, 11.06s/it][2022-12-20 11:52:06,736] [INFO] [timer.py:197:stop] 0/974, RunningAvgSamplesPerSec=5.861897260505879, CurrSamplesPerSec=5.313406336702585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 482/5000 [1:29:47<13:54:26, 11.08s/it][2022-12-20 11:52:17,867] [INFO] [timer.py:197:stop] 0/976, RunningAvgSamplesPerSec=5.861848660793009, CurrSamplesPerSec=5.334826993584506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 483/5000 [1:29:58<13:53:24, 11.07s/it][2022-12-20 11:52:28,900] [INFO] [timer.py:197:stop] 0/978, RunningAvgSamplesPerSec=5.861859052545027, CurrSamplesPerSec=5.31893642772645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 484/5000 [1:30:09<13:52:34, 11.06s/it][2022-12-20 11:52:39,954] [INFO] [logging.py:68:log_dist] [Rank 0] step=490, skipped=3, lr=[9.957609595130721e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:52:39,956] [INFO] [timer.py:197:stop] 0/980, RunningAvgSamplesPerSec=5.861852277267217, CurrSamplesPerSec=5.313113760235646, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 485/5000 [1:30:20<13:51:42, 11.05s/it][2022-12-20 11:52:50,989] [INFO] [timer.py:197:stop] 0/982, RunningAvgSamplesPerSec=5.861860978143288, CurrSamplesPerSec=5.304234243248981, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 486/5000 [1:30:31<13:51:37, 11.05s/it][2022-12-20 11:53:02,045] [INFO] [timer.py:197:stop] 0/984, RunningAvgSamplesPerSec=5.861863179145988, CurrSamplesPerSec=5.318758531130738, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 487/5000 [1:30:42<13:51:43, 11.06s/it][2022-12-20 11:53:13,181] [INFO] [timer.py:197:stop] 0/986, RunningAvgSamplesPerSec=5.861759860676547, CurrSamplesPerSec=5.244360917385617, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 488/5000 [1:30:53<13:53:13, 11.08s/it][2022-12-20 11:53:24,207] [INFO] [timer.py:197:stop] 0/988, RunningAvgSamplesPerSec=5.861777470054748, CurrSamplesPerSec=5.337408422049732, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 489/5000 [1:31:04<13:52:14, 11.07s/it][2022-12-20 11:53:35,294] [INFO] [timer.py:197:stop] 0/990, RunningAvgSamplesPerSec=5.861742379124965, CurrSamplesPerSec=5.299526556557946, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 490/5000 [1:31:15<13:51:42, 11.06s/it][2022-12-20 11:53:46,290] [INFO] [timer.py:197:stop] 0/992, RunningAvgSamplesPerSec=5.861790793411443, CurrSamplesPerSec=5.350795299432934, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 491/5000 [1:31:26<13:49:03, 11.03s/it][2022-12-20 11:53:57,268] [INFO] [timer.py:197:stop] 0/994, RunningAvgSamplesPerSec=5.861857972053263, CurrSamplesPerSec=5.330825078497152, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 492/5000 [1:31:37<13:48:04, 11.02s/it][2022-12-20 11:54:07,464] [INFO] [timer.py:197:stop] 0/996, RunningAvgSamplesPerSec=5.862760726188421, CurrSamplesPerSec=6.1398760823761895, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 11:54:18,468] [INFO] [timer.py:197:stop] 0/998, RunningAvgSamplesPerSec=5.862806601468242, CurrSamplesPerSec=5.3282576274830635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 493/5000 [1:31:54<15:46:10, 12.60s/it][2022-12-20 11:54:29,490] [INFO] [logging.py:68:log_dist] [Rank 0] step=500, skipped=3, lr=[9.990316248055788e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:54:29,492] [INFO] [timer.py:197:stop] 0/1000, RunningAvgSamplesPerSec=5.8628258580253645, CurrSamplesPerSec=5.298601835505908, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 494/5000 [1:32:05<15:10:31, 12.12s/it][2022-12-20 11:54:40,492] [INFO] [timer.py:197:stop] 0/1002, RunningAvgSamplesPerSec=5.862868840592858, CurrSamplesPerSec=5.306693402753534, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 495/5000 [1:32:16<14:45:00, 11.79s/it][2022-12-20 11:54:51,535] [INFO] [timer.py:197:stop] 0/1004, RunningAvgSamplesPerSec=5.8628660971762745, CurrSamplesPerSec=5.305520372365131, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 496/5000 [1:32:27<14:28:02, 11.56s/it][2022-12-20 11:55:02,522] [INFO] [timer.py:197:stop] 0/1006, RunningAvgSamplesPerSec=5.862935660536091, CurrSamplesPerSec=5.3474459636657565, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 497/5000 [1:32:38<14:14:52, 11.39s/it][2022-12-20 11:55:13,651] [INFO] [timer.py:197:stop] 0/1008, RunningAvgSamplesPerSec=5.8628397577773335, CurrSamplesPerSec=5.314844236630596, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 498/5000 [1:32:49<14:08:48, 11.31s/it][2022-12-20 11:55:24,687] [INFO] [timer.py:197:stop] 0/1010, RunningAvgSamplesPerSec=5.862843990873679, CurrSamplesPerSec=5.308032155149343, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|▉         | 499/5000 [1:33:00<14:02:23, 11.23s/it][2022-12-20 11:55:35,763] [INFO] [timer.py:197:stop] 0/1012, RunningAvgSamplesPerSec=5.862806177908336, CurrSamplesPerSec=5.284332076009526, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 500/5000 [1:33:11<13:58:44, 11.18s/it]
                                                       
{'loss': 0.0053, 'learning_rate': 9.995555555555556e-06, 'epoch': 12.19}

 10%|█         | 500/5000 [1:33:11<13:58:44, 11.18s/it][2022-12-20 11:55:46,754] [INFO] [timer.py:197:stop] 0/1014, RunningAvgSamplesPerSec=5.862859544268758, CurrSamplesPerSec=5.350377016221779, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 501/5000 [1:33:22<13:54:14, 11.13s/it][2022-12-20 11:55:57,799] [INFO] [timer.py:197:stop] 0/1016, RunningAvgSamplesPerSec=5.8628535847549, CurrSamplesPerSec=5.319512143888462, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 502/5000 [1:33:33<13:52:14, 11.10s/it][2022-12-20 11:56:08,896] [INFO] [timer.py:197:stop] 0/1018, RunningAvgSamplesPerSec=5.862812160358887, CurrSamplesPerSec=5.30030675323902, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 503/5000 [1:33:44<13:51:57, 11.10s/it][2022-12-20 11:56:19,983] [INFO] [logging.py:68:log_dist] [Rank 0] step=510, skipped=3, lr=[9.986666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:56:19,985] [INFO] [timer.py:197:stop] 0/1020, RunningAvgSamplesPerSec=5.862760853128505, CurrSamplesPerSec=5.289827633515231, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 504/5000 [1:33:55<13:51:30, 11.10s/it][2022-12-20 11:56:31,031] [INFO] [timer.py:197:stop] 0/1022, RunningAvgSamplesPerSec=5.862752882088729, CurrSamplesPerSec=5.272654846689721, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 505/5000 [1:34:06<13:50:11, 11.08s/it][2022-12-20 11:56:41,980] [INFO] [timer.py:197:stop] 0/1024, RunningAvgSamplesPerSec=5.8628591346910905, CurrSamplesPerSec=5.340877805754215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 506/5000 [1:34:17<13:47:01, 11.04s/it][2022-12-20 11:56:52,996] [INFO] [timer.py:197:stop] 0/1026, RunningAvgSamplesPerSec=5.862883771659825, CurrSamplesPerSec=5.3432242911499035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 507/5000 [1:34:28<13:46:16, 11.03s/it][2022-12-20 11:57:04,098] [INFO] [timer.py:197:stop] 0/1028, RunningAvgSamplesPerSec=5.862818306597748, CurrSamplesPerSec=5.267980693320698, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 508/5000 [1:34:39<13:47:36, 11.05s/it][2022-12-20 11:57:15,089] [INFO] [timer.py:197:stop] 0/1030, RunningAvgSamplesPerSec=5.862867926374902, CurrSamplesPerSec=5.366397061490027, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 509/5000 [1:34:50<13:46:01, 11.04s/it][2022-12-20 11:57:26,122] [INFO] [timer.py:197:stop] 0/1032, RunningAvgSamplesPerSec=5.862893639332106, CurrSamplesPerSec=5.33695042238252, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 510/5000 [1:35:01<13:45:45, 11.03s/it][2022-12-20 11:57:37,200] [INFO] [timer.py:197:stop] 0/1034, RunningAvgSamplesPerSec=5.862853608084702, CurrSamplesPerSec=5.349240878776147, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 511/5000 [1:35:12<13:46:32, 11.05s/it][2022-12-20 11:57:48,222] [INFO] [timer.py:197:stop] 0/1036, RunningAvgSamplesPerSec=5.862870680158577, CurrSamplesPerSec=5.336007292587233, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 512/5000 [1:35:23<13:45:46, 11.04s/it][2022-12-20 11:57:59,367] [INFO] [timer.py:197:stop] 0/1038, RunningAvgSamplesPerSec=5.862760913883875, CurrSamplesPerSec=5.239424640382673, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 513/5000 [1:35:34<13:47:58, 11.07s/it][2022-12-20 11:58:10,413] [INFO] [logging.py:68:log_dist] [Rank 0] step=520, skipped=3, lr=[9.964444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 11:58:10,414] [INFO] [timer.py:197:stop] 0/1040, RunningAvgSamplesPerSec=5.862770095965173, CurrSamplesPerSec=5.328816745234495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 514/5000 [1:35:46<13:47:13, 11.06s/it][2022-12-20 11:58:21,442] [INFO] [timer.py:197:stop] 0/1042, RunningAvgSamplesPerSec=5.8627807396708365, CurrSamplesPerSec=5.307206451888165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 515/5000 [1:35:57<13:46:14, 11.05s/it][2022-12-20 11:58:32,616] [INFO] [timer.py:197:stop] 0/1044, RunningAvgSamplesPerSec=5.862712578317147, CurrSamplesPerSec=5.325071226314597, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 516/5000 [1:36:08<13:48:45, 11.09s/it][2022-12-20 11:58:43,644] [INFO] [timer.py:197:stop] 0/1046, RunningAvgSamplesPerSec=5.862723290730008, CurrSamplesPerSec=5.344812250634921, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 517/5000 [1:36:19<13:47:11, 11.07s/it][2022-12-20 11:58:54,923] [INFO] [timer.py:197:stop] 0/1048, RunningAvgSamplesPerSec=5.862478578190851, CurrSamplesPerSec=5.111487418890134, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 518/5000 [1:36:30<13:51:39, 11.13s/it][2022-12-20 11:59:05,924] [INFO] [timer.py:197:stop] 0/1050, RunningAvgSamplesPerSec=5.862520557562685, CurrSamplesPerSec=5.3520526757446545, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 519/5000 [1:36:41<13:48:30, 11.09s/it][2022-12-20 11:59:16,954] [INFO] [timer.py:197:stop] 0/1052, RunningAvgSamplesPerSec=5.862530847784786, CurrSamplesPerSec=5.3358861632905805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 520/5000 [1:36:52<13:46:53, 11.07s/it][2022-12-20 11:59:28,143] [INFO] [timer.py:197:stop] 0/1054, RunningAvgSamplesPerSec=5.8623777361711, CurrSamplesPerSec=5.152182666891972, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 521/5000 [1:37:03<13:49:16, 11.11s/it][2022-12-20 11:59:39,237] [INFO] [timer.py:197:stop] 0/1056, RunningAvgSamplesPerSec=5.862322413396331, CurrSamplesPerSec=5.276953453353096, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 522/5000 [1:37:14<13:48:46, 11.10s/it][2022-12-20 11:59:50,487] [INFO] [timer.py:197:stop] 0/1058, RunningAvgSamplesPerSec=5.862119384624662, CurrSamplesPerSec=5.151062907317255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 523/5000 [1:37:26<13:52:12, 11.15s/it][2022-12-20 12:00:01,776] [INFO] [logging.py:68:log_dist] [Rank 0] step=530, skipped=3, lr=[9.942222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:00:01,777] [INFO] [timer.py:197:stop] 0/1060, RunningAvgSamplesPerSec=5.862099789751432, CurrSamplesPerSec=5.309153163473421, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 524/5000 [1:37:37<13:54:43, 11.19s/it][2022-12-20 12:00:12,784] [INFO] [timer.py:197:stop] 0/1062, RunningAvgSamplesPerSec=5.8621335909239916, CurrSamplesPerSec=5.317789372355522, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 10%|█         | 525/5000 [1:37:48<13:50:27, 11.13s/it]
                                                       
{'loss': 0.0038, 'learning_rate': 9.940000000000001e-06, 'epoch': 12.8}

 10%|█         | 525/5000 [1:37:48<13:50:27, 11.13s/it][2022-12-20 12:00:23,927] [INFO] [timer.py:197:stop] 0/1064, RunningAvgSamplesPerSec=5.862031584392509, CurrSamplesPerSec=5.198011213905655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 526/5000 [1:37:59<13:50:27, 11.14s/it][2022-12-20 12:00:34,912] [INFO] [timer.py:197:stop] 0/1066, RunningAvgSamplesPerSec=5.862089365515606, CurrSamplesPerSec=5.349458984853824, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 527/5000 [1:38:10<13:46:52, 11.09s/it][2022-12-20 12:00:46,014] [INFO] [timer.py:197:stop] 0/1068, RunningAvgSamplesPerSec=5.862028070256045, CurrSamplesPerSec=5.2803342773339725, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 528/5000 [1:38:21<13:46:55, 11.09s/it][2022-12-20 12:00:57,152] [INFO] [timer.py:197:stop] 0/1070, RunningAvgSamplesPerSec=5.861991045135431, CurrSamplesPerSec=5.323989524629343, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 529/5000 [1:38:32<13:47:41, 11.11s/it][2022-12-20 12:01:08,193] [INFO] [timer.py:197:stop] 0/1072, RunningAvgSamplesPerSec=5.8619906224371245, CurrSamplesPerSec=5.29884532802666, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 530/5000 [1:38:43<13:46:02, 11.09s/it][2022-12-20 12:01:19,328] [INFO] [timer.py:197:stop] 0/1074, RunningAvgSamplesPerSec=5.86189695067797, CurrSamplesPerSec=5.214814722717873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 531/5000 [1:38:54<13:46:54, 11.10s/it][2022-12-20 12:01:30,372] [INFO] [timer.py:197:stop] 0/1076, RunningAvgSamplesPerSec=5.861895021616079, CurrSamplesPerSec=5.310112453880426, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 532/5000 [1:39:05<13:45:25, 11.08s/it][2022-12-20 12:01:41,407] [INFO] [timer.py:197:stop] 0/1078, RunningAvgSamplesPerSec=5.8619119768703465, CurrSamplesPerSec=5.330239502254747, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 533/5000 [1:39:17<13:44:07, 11.07s/it][2022-12-20 12:01:51,485] [INFO] [logging.py:68:log_dist] [Rank 0] step=540, skipped=3, lr=[9.920000000000002e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:01:51,487] [INFO] [timer.py:197:stop] 0/1080, RunningAvgSamplesPerSec=5.862868034264494, CurrSamplesPerSec=5.361618272602119, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 534/5000 [1:39:31<15:11:25, 12.24s/it][2022-12-20 12:02:02,490] [INFO] [timer.py:197:stop] 0/1082, RunningAvgSamplesPerSec=5.862904120844008, CurrSamplesPerSec=5.304934680650925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 535/5000 [1:39:43<14:45:02, 11.89s/it][2022-12-20 12:02:13,564] [INFO] [timer.py:197:stop] 0/1084, RunningAvgSamplesPerSec=5.862881204107676, CurrSamplesPerSec=5.303662248282692, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 536/5000 [1:39:54<14:25:33, 11.63s/it][2022-12-20 12:02:24,593] [INFO] [timer.py:197:stop] 0/1086, RunningAvgSamplesPerSec=5.862893707693901, CurrSamplesPerSec=5.304491670755829, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 537/5000 [1:40:05<14:11:50, 11.45s/it][2022-12-20 12:02:35,618] [INFO] [timer.py:197:stop] 0/1088, RunningAvgSamplesPerSec=5.86290823132556, CurrSamplesPerSec=5.306093397521995, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 538/5000 [1:40:16<14:02:42, 11.33s/it][2022-12-20 12:02:46,829] [INFO] [timer.py:197:stop] 0/1090, RunningAvgSamplesPerSec=5.8628966505737985, CurrSamplesPerSec=5.304435068102401, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 539/5000 [1:40:27<13:59:12, 11.29s/it][2022-12-20 12:02:57,791] [INFO] [timer.py:197:stop] 0/1092, RunningAvgSamplesPerSec=5.862972269149122, CurrSamplesPerSec=5.361025912103809, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 540/5000 [1:40:38<13:51:20, 11.18s/it][2022-12-20 12:03:08,792] [INFO] [timer.py:197:stop] 0/1094, RunningAvgSamplesPerSec=5.8630085301258035, CurrSamplesPerSec=5.308453291365968, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 541/5000 [1:40:49<13:49:59, 11.17s/it][2022-12-20 12:03:19,880] [INFO] [timer.py:197:stop] 0/1096, RunningAvgSamplesPerSec=5.862963941668028, CurrSamplesPerSec=5.350500723976222, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 542/5000 [1:41:00<13:45:05, 11.10s/it][2022-12-20 12:03:30,897] [INFO] [timer.py:197:stop] 0/1098, RunningAvgSamplesPerSec=5.86298569572506, CurrSamplesPerSec=5.295404830925499, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 543/5000 [1:41:11<13:44:23, 11.10s/it][2022-12-20 12:03:42,027] [INFO] [logging.py:68:log_dist] [Rank 0] step=550, skipped=3, lr=[9.89777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:03:42,029] [INFO] [timer.py:197:stop] 0/1100, RunningAvgSamplesPerSec=5.862893786708347, CurrSamplesPerSec=5.250315143550748, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 544/5000 [1:41:22<13:44:39, 11.10s/it][2022-12-20 12:03:53,090] [INFO] [timer.py:197:stop] 0/1102, RunningAvgSamplesPerSec=5.8628717776715975, CurrSamplesPerSec=5.300353639325799, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 545/5000 [1:41:33<13:42:37, 11.08s/it][2022-12-20 12:04:04,088] [INFO] [timer.py:197:stop] 0/1104, RunningAvgSamplesPerSec=5.862927963136141, CurrSamplesPerSec=5.321519587187093, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 546/5000 [1:41:44<13:43:02, 11.09s/it][2022-12-20 12:04:15,205] [INFO] [timer.py:197:stop] 0/1106, RunningAvgSamplesPerSec=5.862852172047862, CurrSamplesPerSec=5.311292350082061, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 547/5000 [1:41:55<13:42:00, 11.08s/it][2022-12-20 12:04:26,306] [INFO] [timer.py:197:stop] 0/1108, RunningAvgSamplesPerSec=5.862810110387553, CurrSamplesPerSec=5.281787794621473, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 548/5000 [1:42:06<13:41:50, 11.08s/it][2022-12-20 12:04:37,361] [INFO] [timer.py:197:stop] 0/1110, RunningAvgSamplesPerSec=5.862794755789113, CurrSamplesPerSec=5.284954430573337, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 549/5000 [1:42:17<13:42:39, 11.09s/it][2022-12-20 12:04:48,428] [INFO] [timer.py:197:stop] 0/1112, RunningAvgSamplesPerSec=5.862784069735843, CurrSamplesPerSec=5.332795303373525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 550/5000 [1:42:28<13:41:23, 11.07s/it]
                                                       
{'loss': 0.0038, 'learning_rate': 9.884444444444445e-06, 'epoch': 13.41}

 11%|█         | 550/5000 [1:42:29<13:41:23, 11.07s/it][2022-12-20 12:04:59,489] [INFO] [timer.py:197:stop] 0/1114, RunningAvgSamplesPerSec=5.862763875141886, CurrSamplesPerSec=5.314834344991809, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 551/5000 [1:42:40<13:41:05, 11.07s/it][2022-12-20 12:05:10,566] [INFO] [timer.py:197:stop] 0/1116, RunningAvgSamplesPerSec=5.862728669007944, CurrSamplesPerSec=5.308764252861252, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 552/5000 [1:42:51<13:41:01, 11.07s/it][2022-12-20 12:05:21,624] [INFO] [timer.py:197:stop] 0/1118, RunningAvgSamplesPerSec=5.862711699076243, CurrSamplesPerSec=5.3262347414799835, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 553/5000 [1:43:02<13:40:14, 11.07s/it][2022-12-20 12:05:32,679] [INFO] [logging.py:68:log_dist] [Rank 0] step=560, skipped=3, lr=[9.875555555555557e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:05:32,680] [INFO] [timer.py:197:stop] 0/1120, RunningAvgSamplesPerSec=5.862695236159943, CurrSamplesPerSec=5.318703520492889, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 554/5000 [1:43:13<13:40:29, 11.07s/it][2022-12-20 12:05:43,725] [INFO] [timer.py:197:stop] 0/1122, RunningAvgSamplesPerSec=5.862695806887015, CurrSamplesPerSec=5.355648621046771, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 555/5000 [1:43:24<13:38:00, 11.04s/it][2022-12-20 12:05:54,736] [INFO] [timer.py:197:stop] 0/1124, RunningAvgSamplesPerSec=5.862724658095585, CurrSamplesPerSec=5.3196257840308725, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 556/5000 [1:43:35<13:36:55, 11.03s/it][2022-12-20 12:06:05,693] [INFO] [timer.py:197:stop] 0/1126, RunningAvgSamplesPerSec=5.86280257965277, CurrSamplesPerSec=5.357422316959522, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 557/5000 [1:43:46<13:35:55, 11.02s/it][2022-12-20 12:06:16,706] [INFO] [timer.py:197:stop] 0/1128, RunningAvgSamplesPerSec=5.8628285696932405, CurrSamplesPerSec=5.341244228527144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 558/5000 [1:43:57<13:35:10, 11.01s/it][2022-12-20 12:06:27,752] [INFO] [timer.py:197:stop] 0/1130, RunningAvgSamplesPerSec=5.862822780226703, CurrSamplesPerSec=5.294154506807159, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 559/5000 [1:44:08<13:35:05, 11.01s/it][2022-12-20 12:06:38,754] [INFO] [timer.py:197:stop] 0/1132, RunningAvgSamplesPerSec=5.862857750756119, CurrSamplesPerSec=5.3054606019795605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 560/5000 [1:44:19<13:35:54, 11.03s/it][2022-12-20 12:06:49,796] [INFO] [timer.py:197:stop] 0/1134, RunningAvgSamplesPerSec=5.862855272544059, CurrSamplesPerSec=5.3194039899692935, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 561/5000 [1:44:30<13:35:32, 11.02s/it][2022-12-20 12:07:00,814] [INFO] [timer.py:197:stop] 0/1136, RunningAvgSamplesPerSec=5.862877056260085, CurrSamplesPerSec=5.319090937624736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█         | 562/5000 [1:44:41<13:35:11, 11.02s/it][2022-12-20 12:07:11,849] [INFO] [timer.py:197:stop] 0/1138, RunningAvgSamplesPerSec=5.862882328466611, CurrSamplesPerSec=5.30234015249839, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 563/5000 [1:44:52<13:35:49, 11.03s/it][2022-12-20 12:07:22,905] [INFO] [logging.py:68:log_dist] [Rank 0] step=570, skipped=3, lr=[9.853333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:07:22,906] [INFO] [timer.py:197:stop] 0/1140, RunningAvgSamplesPerSec=5.862865946098857, CurrSamplesPerSec=5.302454735975793, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 564/5000 [1:45:03<13:36:13, 11.04s/it][2022-12-20 12:07:33,939] [INFO] [timer.py:197:stop] 0/1142, RunningAvgSamplesPerSec=5.86287412108049, CurrSamplesPerSec=5.326575059031055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 565/5000 [1:45:14<13:36:01, 11.04s/it][2022-12-20 12:07:45,002] [INFO] [timer.py:197:stop] 0/1144, RunningAvgSamplesPerSec=5.862852492976112, CurrSamplesPerSec=5.30425604395865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 566/5000 [1:45:25<13:35:51, 11.04s/it][2022-12-20 12:07:56,011] [INFO] [timer.py:197:stop] 0/1146, RunningAvgSamplesPerSec=5.86287993711664, CurrSamplesPerSec=5.331229086775822, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 567/5000 [1:45:36<13:34:49, 11.03s/it][2022-12-20 12:08:07,080] [INFO] [timer.py:197:stop] 0/1148, RunningAvgSamplesPerSec=5.862851061401254, CurrSamplesPerSec=5.272555010587322, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 568/5000 [1:45:47<13:35:54, 11.05s/it][2022-12-20 12:08:18,120] [INFO] [timer.py:197:stop] 0/1150, RunningAvgSamplesPerSec=5.862849031382978, CurrSamplesPerSec=5.311940412808317, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 569/5000 [1:45:58<13:35:51, 11.05s/it][2022-12-20 12:08:29,377] [INFO] [timer.py:197:stop] 0/1152, RunningAvgSamplesPerSec=5.862652479474226, CurrSamplesPerSec=5.144166065475358, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 570/5000 [1:46:10<13:43:05, 11.15s/it][2022-12-20 12:08:40,554] [INFO] [timer.py:197:stop] 0/1154, RunningAvgSamplesPerSec=5.862664334838466, CurrSamplesPerSec=5.312503891330501, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 571/5000 [1:46:21<13:40:01, 11.11s/it][2022-12-20 12:08:51,574] [INFO] [timer.py:197:stop] 0/1156, RunningAvgSamplesPerSec=5.8626824267650735, CurrSamplesPerSec=5.310558504377071, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 572/5000 [1:46:32<13:38:35, 11.09s/it][2022-12-20 12:09:02,634] [INFO] [timer.py:197:stop] 0/1158, RunningAvgSamplesPerSec=5.862664296707742, CurrSamplesPerSec=5.304899664871065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 573/5000 [1:46:43<13:36:27, 11.07s/it][2022-12-20 12:09:13,605] [INFO] [logging.py:68:log_dist] [Rank 0] step=580, skipped=3, lr=[9.831111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:09:13,607] [INFO] [timer.py:197:stop] 0/1160, RunningAvgSamplesPerSec=5.862725159592587, CurrSamplesPerSec=5.331008018055988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 11%|█▏        | 574/5000 [1:46:54<13:34:52, 11.05s/it][2022-12-20 12:09:23,712] [INFO] [timer.py:197:stop] 0/1162, RunningAvgSamplesPerSec=5.863580500624223, CurrSamplesPerSec=6.255159265519461, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 12:09:34,755] [INFO] [timer.py:197:stop] 0/1164, RunningAvgSamplesPerSec=5.863578122482712, CurrSamplesPerSec=5.332878787318913, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 575/5000 [1:47:10<15:28:58, 12.60s/it]
                                                       
{'loss': 0.004, 'learning_rate': 9.826666666666667e-06, 'epoch': 14.02}

 12%|█▏        | 575/5000 [1:47:10<15:28:58, 12.60s/it][2022-12-20 12:09:45,718] [INFO] [timer.py:197:stop] 0/1166, RunningAvgSamplesPerSec=5.863647840199046, CurrSamplesPerSec=5.341744846693069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 576/5000 [1:47:21<14:52:37, 12.11s/it][2022-12-20 12:09:56,780] [INFO] [timer.py:197:stop] 0/1168, RunningAvgSamplesPerSec=5.863624610582614, CurrSamplesPerSec=5.286939828272781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 577/5000 [1:47:32<14:29:21, 11.79s/it][2022-12-20 12:10:07,807] [INFO] [timer.py:197:stop] 0/1170, RunningAvgSamplesPerSec=5.863650874337149, CurrSamplesPerSec=5.33207626309121, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 578/5000 [1:47:43<14:12:11, 11.56s/it][2022-12-20 12:10:18,838] [INFO] [timer.py:197:stop] 0/1172, RunningAvgSamplesPerSec=5.863656636364203, CurrSamplesPerSec=5.303352303287028, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 579/5000 [1:47:54<14:00:14, 11.40s/it][2022-12-20 12:10:29,905] [INFO] [timer.py:197:stop] 0/1174, RunningAvgSamplesPerSec=5.863631482891206, CurrSamplesPerSec=5.29866270653814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 580/5000 [1:48:05<13:52:37, 11.30s/it][2022-12-20 12:10:40,944] [INFO] [timer.py:197:stop] 0/1176, RunningAvgSamplesPerSec=5.863629878306914, CurrSamplesPerSec=5.316259330743629, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 581/5000 [1:48:16<13:46:36, 11.22s/it][2022-12-20 12:10:52,372] [INFO] [timer.py:197:stop] 0/1178, RunningAvgSamplesPerSec=5.863273524329394, CurrSamplesPerSec=4.97460628430686, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 582/5000 [1:48:27<13:50:56, 11.28s/it][2022-12-20 12:11:03,397] [INFO] [logging.py:68:log_dist] [Rank 0] step=590, skipped=3, lr=[9.80888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:11:03,399] [INFO] [timer.py:197:stop] 0/1180, RunningAvgSamplesPerSec=5.863284195555395, CurrSamplesPerSec=5.321181603005539, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 583/5000 [1:48:38<13:45:03, 11.21s/it][2022-12-20 12:11:14,404] [INFO] [timer.py:197:stop] 0/1182, RunningAvgSamplesPerSec=5.863314604112909, CurrSamplesPerSec=5.34898207411068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 584/5000 [1:48:49<13:40:24, 11.15s/it][2022-12-20 12:11:25,532] [INFO] [timer.py:197:stop] 0/1184, RunningAvgSamplesPerSec=5.863232940620662, CurrSamplesPerSec=5.246413951647742, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 585/5000 [1:49:01<13:39:47, 11.14s/it][2022-12-20 12:11:36,539] [INFO] [timer.py:197:stop] 0/1186, RunningAvgSamplesPerSec=5.86326121786836, CurrSamplesPerSec=5.317460288532767, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 586/5000 [1:49:12<13:36:39, 11.10s/it][2022-12-20 12:11:47,538] [INFO] [timer.py:197:stop] 0/1188, RunningAvgSamplesPerSec=5.86331189655806, CurrSamplesPerSec=5.3440080460097334, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 587/5000 [1:49:23<13:34:13, 11.07s/it][2022-12-20 12:11:58,567] [INFO] [timer.py:197:stop] 0/1190, RunningAvgSamplesPerSec=5.863320360711068, CurrSamplesPerSec=5.367243218532154, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 588/5000 [1:49:34<13:33:07, 11.06s/it][2022-12-20 12:12:09,597] [INFO] [timer.py:197:stop] 0/1192, RunningAvgSamplesPerSec=5.863330206503183, CurrSamplesPerSec=5.318809748954429, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 589/5000 [1:49:45<13:32:20, 11.05s/it][2022-12-20 12:12:20,569] [INFO] [timer.py:197:stop] 0/1194, RunningAvgSamplesPerSec=5.8633902806505835, CurrSamplesPerSec=5.33447968427867, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 590/5000 [1:49:56<13:30:25, 11.03s/it][2022-12-20 12:12:31,595] [INFO] [timer.py:197:stop] 0/1196, RunningAvgSamplesPerSec=5.863401905826546, CurrSamplesPerSec=5.304328155275038, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 591/5000 [1:50:07<13:30:14, 11.03s/it][2022-12-20 12:12:42,635] [INFO] [timer.py:197:stop] 0/1198, RunningAvgSamplesPerSec=5.863400404656514, CurrSamplesPerSec=5.312666228963957, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 592/5000 [1:50:18<13:30:22, 11.03s/it][2022-12-20 12:12:53,688] [INFO] [logging.py:68:log_dist] [Rank 0] step=600, skipped=3, lr=[9.786666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:12:53,690] [INFO] [timer.py:197:stop] 0/1200, RunningAvgSamplesPerSec=5.863400468258839, CurrSamplesPerSec=5.316812984493476, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 593/5000 [1:50:29<13:30:43, 11.04s/it][2022-12-20 12:13:04,674] [INFO] [timer.py:197:stop] 0/1202, RunningAvgSamplesPerSec=5.863449797238821, CurrSamplesPerSec=5.321397637881127, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 594/5000 [1:50:40<13:29:20, 11.02s/it][2022-12-20 12:13:16,123] [INFO] [timer.py:197:stop] 0/1204, RunningAvgSamplesPerSec=5.863091427867219, CurrSamplesPerSec=5.004240291117342, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 595/5000 [1:50:51<13:38:56, 11.15s/it][2022-12-20 12:13:27,119] [INFO] [timer.py:197:stop] 0/1206, RunningAvgSamplesPerSec=5.8631588952166505, CurrSamplesPerSec=5.363048529538043, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 596/5000 [1:51:02<13:34:54, 11.10s/it][2022-12-20 12:13:38,159] [INFO] [timer.py:197:stop] 0/1208, RunningAvgSamplesPerSec=5.863158124982087, CurrSamplesPerSec=5.310512908400964, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 597/5000 [1:51:13<13:33:21, 11.08s/it][2022-12-20 12:13:49,287] [INFO] [timer.py:197:stop] 0/1210, RunningAvgSamplesPerSec=5.863079748411406, CurrSamplesPerSec=5.255263124304073, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 598/5000 [1:51:24<13:34:08, 11.10s/it][2022-12-20 12:14:00,340] [INFO] [timer.py:197:stop] 0/1212, RunningAvgSamplesPerSec=5.863067960675532, CurrSamplesPerSec=5.2852907419843715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 599/5000 [1:51:35<13:32:59, 11.08s/it][2022-12-20 12:14:11,409] [INFO] [timer.py:197:stop] 0/1214, RunningAvgSamplesPerSec=5.863042127607864, CurrSamplesPerSec=5.302098433215638, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 600/5000 [1:51:47<13:32:28, 11.08s/it]
                                                       
{'loss': 0.0037, 'learning_rate': 9.771111111111113e-06, 'epoch': 14.63}

 12%|█▏        | 600/5000 [1:51:47<13:32:28, 11.08s/it][2022-12-20 12:14:22,513] [INFO] [timer.py:197:stop] 0/1216, RunningAvgSamplesPerSec=5.862987125928283, CurrSamplesPerSec=5.301469102955862, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 601/5000 [1:51:58<13:32:50, 11.09s/it][2022-12-20 12:14:33,514] [INFO] [timer.py:197:stop] 0/1218, RunningAvgSamplesPerSec=5.863037146235074, CurrSamplesPerSec=5.338541449353022, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 602/5000 [1:52:09<13:30:46, 11.06s/it][2022-12-20 12:14:44,778] [INFO] [logging.py:68:log_dist] [Rank 0] step=610, skipped=3, lr=[9.764444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:14:44,779] [INFO] [timer.py:197:stop] 0/1220, RunningAvgSamplesPerSec=5.862837880211341, CurrSamplesPerSec=5.127814643669685, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 603/5000 [1:52:20<13:35:04, 11.12s/it][2022-12-20 12:14:55,870] [INFO] [timer.py:197:stop] 0/1222, RunningAvgSamplesPerSec=5.862792487872289, CurrSamplesPerSec=5.29011952750889, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 604/5000 [1:52:31<13:34:13, 11.11s/it][2022-12-20 12:15:06,971] [INFO] [timer.py:197:stop] 0/1224, RunningAvgSamplesPerSec=5.862754667241762, CurrSamplesPerSec=5.287610916331852, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 605/5000 [1:52:42<13:33:45, 11.11s/it][2022-12-20 12:15:18,249] [INFO] [timer.py:197:stop] 0/1226, RunningAvgSamplesPerSec=5.862546627192508, CurrSamplesPerSec=5.272478996837127, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 606/5000 [1:52:53<13:37:16, 11.16s/it][2022-12-20 12:15:29,256] [INFO] [timer.py:197:stop] 0/1228, RunningAvgSamplesPerSec=5.862576500453656, CurrSamplesPerSec=5.331880541448883, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 607/5000 [1:53:04<13:33:43, 11.11s/it][2022-12-20 12:15:40,285] [INFO] [timer.py:197:stop] 0/1230, RunningAvgSamplesPerSec=5.8625861542527975, CurrSamplesPerSec=5.297582087664745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 608/5000 [1:53:15<13:31:40, 11.09s/it][2022-12-20 12:15:51,283] [INFO] [timer.py:197:stop] 0/1232, RunningAvgSamplesPerSec=5.862623279244614, CurrSamplesPerSec=5.3209290922857475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 609/5000 [1:53:26<13:29:31, 11.06s/it][2022-12-20 12:16:02,299] [INFO] [timer.py:197:stop] 0/1234, RunningAvgSamplesPerSec=5.862661647724114, CurrSamplesPerSec=5.36635071618725, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 610/5000 [1:53:37<13:28:18, 11.05s/it][2022-12-20 12:16:13,383] [INFO] [timer.py:197:stop] 0/1236, RunningAvgSamplesPerSec=5.862624044195371, CurrSamplesPerSec=5.312124581443448, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 611/5000 [1:53:48<13:28:56, 11.06s/it][2022-12-20 12:16:24,443] [INFO] [timer.py:197:stop] 0/1238, RunningAvgSamplesPerSec=5.8626069911780325, CurrSamplesPerSec=5.282982792189083, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 612/5000 [1:54:00<13:28:47, 11.06s/it][2022-12-20 12:16:35,440] [INFO] [logging.py:68:log_dist] [Rank 0] step=620, skipped=3, lr=[9.742222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:16:35,442] [INFO] [timer.py:197:stop] 0/1240, RunningAvgSamplesPerSec=5.8626431637747505, CurrSamplesPerSec=5.314945049261884, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 613/5000 [1:54:11<13:27:16, 11.04s/it][2022-12-20 12:16:46,466] [INFO] [timer.py:197:stop] 0/1242, RunningAvgSamplesPerSec=5.862656898160635, CurrSamplesPerSec=5.2979451030124745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 614/5000 [1:54:22<13:26:43, 11.04s/it][2022-12-20 12:16:57,526] [INFO] [timer.py:197:stop] 0/1244, RunningAvgSamplesPerSec=5.862639813811557, CurrSamplesPerSec=5.28263409141977, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 615/5000 [1:54:33<13:27:05, 11.04s/it][2022-12-20 12:17:07,756] [INFO] [timer.py:197:stop] 0/1246, RunningAvgSamplesPerSec=5.863358898894835, CurrSamplesPerSec=5.288069653668914, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 616/5000 [1:54:48<14:56:50, 12.27s/it][2022-12-20 12:17:18,767] [INFO] [timer.py:197:stop] 0/1248, RunningAvgSamplesPerSec=5.863381114901359, CurrSamplesPerSec=5.306090041231018, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 617/5000 [1:54:59<14:28:41, 11.89s/it][2022-12-20 12:17:30,019] [INFO] [timer.py:197:stop] 0/1250, RunningAvgSamplesPerSec=5.863197154396863, CurrSamplesPerSec=5.093768483582376, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 618/5000 [1:55:10<14:15:51, 11.72s/it][2022-12-20 12:17:41,084] [INFO] [timer.py:197:stop] 0/1252, RunningAvgSamplesPerSec=5.863174761926002, CurrSamplesPerSec=5.304317254601421, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 619/5000 [1:55:21<14:01:21, 11.52s/it][2022-12-20 12:17:52,133] [INFO] [timer.py:197:stop] 0/1254, RunningAvgSamplesPerSec=5.863165212273337, CurrSamplesPerSec=5.3188797271640365, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 620/5000 [1:55:32<13:50:31, 11.38s/it][2022-12-20 12:18:03,252] [INFO] [timer.py:197:stop] 0/1256, RunningAvgSamplesPerSec=5.863096152483799, CurrSamplesPerSec=5.247435635038622, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 621/5000 [1:55:43<13:44:50, 11.30s/it][2022-12-20 12:18:14,344] [INFO] [timer.py:197:stop] 0/1258, RunningAvgSamplesPerSec=5.863051002885076, CurrSamplesPerSec=5.277416153444789, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 622/5000 [1:55:54<13:40:28, 11.24s/it][2022-12-20 12:18:25,385] [INFO] [logging.py:68:log_dist] [Rank 0] step=630, skipped=3, lr=[9.72e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:18:25,387] [INFO] [timer.py:197:stop] 0/1260, RunningAvgSamplesPerSec=5.86305584034761, CurrSamplesPerSec=5.335941954209909, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 623/5000 [1:56:05<13:35:03, 11.17s/it][2022-12-20 12:18:36,389] [INFO] [timer.py:197:stop] 0/1262, RunningAvgSamplesPerSec=5.863087744753293, CurrSamplesPerSec=5.340521845698404, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▏        | 624/5000 [1:56:16<13:30:29, 11.11s/it][2022-12-20 12:18:47,366] [INFO] [timer.py:197:stop] 0/1264, RunningAvgSamplesPerSec=5.863141553277318, CurrSamplesPerSec=5.336999232321179, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 12%|█▎        | 625/5000 [1:56:27<13:28:21, 11.09s/it]
                                                       
{'loss': 0.0047, 'learning_rate': 9.715555555555557e-06, 'epoch': 15.24}

 12%|█▎        | 625/5000 [1:56:27<13:28:21, 11.09s/it][2022-12-20 12:18:58,453] [INFO] [timer.py:197:stop] 0/1266, RunningAvgSamplesPerSec=5.863102153890871, CurrSamplesPerSec=5.282664447558434, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 626/5000 [1:56:39<13:28:03, 11.08s/it][2022-12-20 12:19:09,474] [INFO] [timer.py:197:stop] 0/1268, RunningAvgSamplesPerSec=5.863121883427295, CurrSamplesPerSec=5.332679192993292, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 627/5000 [1:56:50<13:26:47, 11.07s/it][2022-12-20 12:19:20,555] [INFO] [timer.py:197:stop] 0/1270, RunningAvgSamplesPerSec=5.863086660569417, CurrSamplesPerSec=5.292333966673425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 628/5000 [1:57:01<13:27:27, 11.08s/it][2022-12-20 12:19:31,638] [INFO] [timer.py:197:stop] 0/1272, RunningAvgSamplesPerSec=5.863051895937327, CurrSamplesPerSec=5.316958103388103, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 629/5000 [1:57:12<13:26:53, 11.08s/it][2022-12-20 12:19:42,705] [INFO] [timer.py:197:stop] 0/1274, RunningAvgSamplesPerSec=5.863028834425083, CurrSamplesPerSec=5.312773898773201, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 630/5000 [1:57:23<13:26:14, 11.07s/it][2022-12-20 12:19:53,736] [INFO] [timer.py:197:stop] 0/1276, RunningAvgSamplesPerSec=5.863036113570025, CurrSamplesPerSec=5.334199622227988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 631/5000 [1:57:34<13:26:07, 11.07s/it][2022-12-20 12:20:04,768] [INFO] [timer.py:197:stop] 0/1278, RunningAvgSamplesPerSec=5.863041573892956, CurrSamplesPerSec=5.369767178924936, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 632/5000 [1:57:45<13:23:04, 11.03s/it][2022-12-20 12:20:15,765] [INFO] [logging.py:68:log_dist] [Rank 0] step=640, skipped=3, lr=[9.697777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:20:15,767] [INFO] [timer.py:197:stop] 0/1280, RunningAvgSamplesPerSec=5.863090259390718, CurrSamplesPerSec=5.331522814672224, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 633/5000 [1:57:56<13:21:53, 11.02s/it][2022-12-20 12:20:26,912] [INFO] [timer.py:197:stop] 0/1282, RunningAvgSamplesPerSec=5.863001710927022, CurrSamplesPerSec=5.1802943546958415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 634/5000 [1:58:07<13:25:35, 11.07s/it][2022-12-20 12:20:37,949] [INFO] [timer.py:197:stop] 0/1284, RunningAvgSamplesPerSec=5.863004024664853, CurrSamplesPerSec=5.316416633337824, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 635/5000 [1:58:18<13:24:29, 11.06s/it][2022-12-20 12:20:48,950] [INFO] [timer.py:197:stop] 0/1286, RunningAvgSamplesPerSec=5.863050482418797, CurrSamplesPerSec=5.340652323457854, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 636/5000 [1:58:29<13:23:20, 11.05s/it][2022-12-20 12:20:59,980] [INFO] [timer.py:197:stop] 0/1288, RunningAvgSamplesPerSec=5.863057890252443, CurrSamplesPerSec=5.326588165297044, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 637/5000 [1:58:40<13:22:50, 11.04s/it][2022-12-20 12:21:11,145] [INFO] [timer.py:197:stop] 0/1290, RunningAvgSamplesPerSec=5.862966819186735, CurrSamplesPerSec=5.223858624782511, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 638/5000 [1:58:51<13:25:14, 11.08s/it][2022-12-20 12:21:22,429] [INFO] [timer.py:197:stop] 0/1292, RunningAvgSamplesPerSec=5.862762593629575, CurrSamplesPerSec=5.105547745876969, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 639/5000 [1:59:02<13:29:55, 11.14s/it][2022-12-20 12:21:33,497] [INFO] [timer.py:197:stop] 0/1294, RunningAvgSamplesPerSec=5.862739487045471, CurrSamplesPerSec=5.301280437703114, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 640/5000 [1:59:14<13:27:11, 11.11s/it][2022-12-20 12:21:44,504] [INFO] [timer.py:197:stop] 0/1296, RunningAvgSamplesPerSec=5.862770728218841, CurrSamplesPerSec=5.318178342118792, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 641/5000 [1:59:25<13:25:10, 11.08s/it][2022-12-20 12:21:55,484] [INFO] [timer.py:197:stop] 0/1298, RunningAvgSamplesPerSec=5.862819926252189, CurrSamplesPerSec=5.357801921160977, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 642/5000 [1:59:36<13:23:19, 11.06s/it][2022-12-20 12:22:06,553] [INFO] [logging.py:68:log_dist] [Rank 0] step=650, skipped=3, lr=[9.675555555555555e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:22:06,555] [INFO] [timer.py:197:stop] 0/1300, RunningAvgSamplesPerSec=5.862794355863427, CurrSamplesPerSec=5.301336973051239, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 643/5000 [1:59:47<13:22:27, 11.05s/it][2022-12-20 12:22:17,677] [INFO] [timer.py:197:stop] 0/1302, RunningAvgSamplesPerSec=5.862726185609056, CurrSamplesPerSec=5.220495697218881, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 644/5000 [1:59:58<13:23:50, 11.07s/it][2022-12-20 12:22:28,700] [INFO] [timer.py:197:stop] 0/1304, RunningAvgSamplesPerSec=5.862753436768504, CurrSamplesPerSec=5.305794913996005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 645/5000 [2:00:09<13:22:22, 11.05s/it][2022-12-20 12:22:39,696] [INFO] [timer.py:197:stop] 0/1306, RunningAvgSamplesPerSec=5.862789672329912, CurrSamplesPerSec=5.321486672982884, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 646/5000 [2:00:20<13:21:51, 11.05s/it][2022-12-20 12:22:50,810] [INFO] [timer.py:197:stop] 0/1308, RunningAvgSamplesPerSec=5.862727505102733, CurrSamplesPerSec=5.255740550764884, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 647/5000 [2:00:31<13:23:08, 11.07s/it][2022-12-20 12:23:01,907] [INFO] [timer.py:197:stop] 0/1310, RunningAvgSamplesPerSec=5.862680676010021, CurrSamplesPerSec=5.273880528984437, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 648/5000 [2:00:42<13:23:45, 11.08s/it][2022-12-20 12:23:12,961] [INFO] [timer.py:197:stop] 0/1312, RunningAvgSamplesPerSec=5.8626763088631, CurrSamplesPerSec=5.319816389901659, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 649/5000 [2:00:53<13:24:34, 11.10s/it][2022-12-20 12:23:24,057] [INFO] [timer.py:197:stop] 0/1314, RunningAvgSamplesPerSec=5.862629877647307, CurrSamplesPerSec=5.348313646806721, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 650/5000 [2:01:04<13:22:34, 11.07s/it]
                                                       
{'loss': 0.0047, 'learning_rate': 9.66e-06, 'epoch': 15.84}

 13%|█▎        | 650/5000 [2:01:04<13:22:34, 11.07s/it][2022-12-20 12:23:35,139] [INFO] [timer.py:197:stop] 0/1316, RunningAvgSamplesPerSec=5.8626095502840245, CurrSamplesPerSec=5.30189506257306, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 651/5000 [2:01:15<13:22:27, 11.07s/it][2022-12-20 12:23:46,421] [INFO] [timer.py:197:stop] 0/1318, RunningAvgSamplesPerSec=5.862410658330332, CurrSamplesPerSec=5.109859382071171, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 652/5000 [2:01:26<13:27:08, 11.14s/it][2022-12-20 12:23:57,490] [INFO] [logging.py:68:log_dist] [Rank 0] step=660, skipped=3, lr=[9.653333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:23:57,492] [INFO] [timer.py:197:stop] 0/1320, RunningAvgSamplesPerSec=5.862385090029393, CurrSamplesPerSec=5.298968756321798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 653/5000 [2:01:38<13:25:26, 11.12s/it][2022-12-20 12:24:08,551] [INFO] [timer.py:197:stop] 0/1322, RunningAvgSamplesPerSec=5.862377126324596, CurrSamplesPerSec=5.307546440927752, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 654/5000 [2:01:49<13:23:33, 11.09s/it][2022-12-20 12:24:19,546] [INFO] [timer.py:197:stop] 0/1324, RunningAvgSamplesPerSec=5.86242785843892, CurrSamplesPerSec=5.361750425827344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 655/5000 [2:02:00<13:21:43, 11.07s/it][2022-12-20 12:24:30,589] [INFO] [timer.py:197:stop] 0/1326, RunningAvgSamplesPerSec=5.8624235748933025, CurrSamplesPerSec=5.3237293565647335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 656/5000 [2:02:11<13:20:51, 11.06s/it][2022-12-20 12:24:41,156] [INFO] [timer.py:197:stop] 0/1328, RunningAvgSamplesPerSec=5.862798471516799, CurrSamplesPerSec=5.768034534531889, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 12:24:52,181] [INFO] [timer.py:197:stop] 0/1330, RunningAvgSamplesPerSec=5.86280899818244, CurrSamplesPerSec=5.31184265724856, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 657/5000 [2:02:27<15:21:30, 12.73s/it][2022-12-20 12:25:03,218] [INFO] [timer.py:197:stop] 0/1332, RunningAvgSamplesPerSec=5.862811198224065, CurrSamplesPerSec=5.333165491935825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 658/5000 [2:02:38<14:44:30, 12.22s/it][2022-12-20 12:25:14,265] [INFO] [timer.py:197:stop] 0/1334, RunningAvgSamplesPerSec=5.862804397755479, CurrSamplesPerSec=5.309389015533925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 659/5000 [2:02:49<14:18:48, 11.87s/it][2022-12-20 12:25:25,343] [INFO] [timer.py:197:stop] 0/1336, RunningAvgSamplesPerSec=5.862772639774549, CurrSamplesPerSec=5.2811580813277486, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 660/5000 [2:03:00<14:01:25, 11.63s/it][2022-12-20 12:25:36,351] [INFO] [timer.py:197:stop] 0/1338, RunningAvgSamplesPerSec=5.862799444383477, CurrSamplesPerSec=5.328781836605953, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 661/5000 [2:03:11<13:47:40, 11.45s/it][2022-12-20 12:25:47,509] [INFO] [logging.py:68:log_dist] [Rank 0] step=670, skipped=3, lr=[9.631111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:25:47,511] [INFO] [timer.py:197:stop] 0/1340, RunningAvgSamplesPerSec=5.8627026865352745, CurrSamplesPerSec=5.290824167231978, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 662/5000 [2:03:23<13:41:17, 11.36s/it][2022-12-20 12:25:58,613] [INFO] [timer.py:197:stop] 0/1342, RunningAvgSamplesPerSec=5.862666219141511, CurrSamplesPerSec=5.299673662983459, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 663/5000 [2:03:34<13:35:31, 11.28s/it][2022-12-20 12:26:09,682] [INFO] [timer.py:197:stop] 0/1344, RunningAvgSamplesPerSec=5.8626425559471835, CurrSamplesPerSec=5.292309551013945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 664/5000 [2:03:45<13:30:43, 11.22s/it][2022-12-20 12:26:20,730] [INFO] [timer.py:197:stop] 0/1346, RunningAvgSamplesPerSec=5.86263764821861, CurrSamplesPerSec=5.294762467380563, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 665/5000 [2:03:56<13:26:50, 11.17s/it][2022-12-20 12:26:31,734] [INFO] [timer.py:197:stop] 0/1348, RunningAvgSamplesPerSec=5.862666162095201, CurrSamplesPerSec=5.320942170782142, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 666/5000 [2:04:07<13:23:06, 11.12s/it][2022-12-20 12:26:42,850] [INFO] [timer.py:197:stop] 0/1350, RunningAvgSamplesPerSec=5.8626129566538445, CurrSamplesPerSec=5.283693851879989, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 667/5000 [2:04:18<13:22:52, 11.12s/it][2022-12-20 12:26:53,886] [INFO] [timer.py:197:stop] 0/1352, RunningAvgSamplesPerSec=5.862617422793718, CurrSamplesPerSec=5.302812972207053, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 668/5000 [2:04:29<13:20:54, 11.09s/it][2022-12-20 12:27:05,156] [INFO] [timer.py:197:stop] 0/1354, RunningAvgSamplesPerSec=5.862436879673771, CurrSamplesPerSec=5.120591279213035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 669/5000 [2:04:40<13:25:01, 11.15s/it][2022-12-20 12:27:16,343] [INFO] [timer.py:197:stop] 0/1356, RunningAvgSamplesPerSec=5.862463331594795, CurrSamplesPerSec=5.314263638738938, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 670/5000 [2:04:51<13:25:08, 11.16s/it][2022-12-20 12:27:27,294] [INFO] [timer.py:197:stop] 0/1358, RunningAvgSamplesPerSec=5.862535238866188, CurrSamplesPerSec=5.363107461518593, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 671/5000 [2:05:02<13:20:29, 11.09s/it][2022-12-20 12:27:38,406] [INFO] [logging.py:68:log_dist] [Rank 0] step=680, skipped=3, lr=[9.60888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:27:38,407] [INFO] [timer.py:197:stop] 0/1360, RunningAvgSamplesPerSec=5.862476379374207, CurrSamplesPerSec=5.21719773277008, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 672/5000 [2:05:14<13:20:42, 11.10s/it][2022-12-20 12:27:49,439] [INFO] [timer.py:197:stop] 0/1362, RunningAvgSamplesPerSec=5.862482780361491, CurrSamplesPerSec=5.318447029974287, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 673/5000 [2:05:25<13:19:02, 11.08s/it][2022-12-20 12:28:00,433] [INFO] [timer.py:197:stop] 0/1364, RunningAvgSamplesPerSec=5.86251985997439, CurrSamplesPerSec=5.363120319577375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 13%|█▎        | 674/5000 [2:05:36<13:16:59, 11.05s/it][2022-12-20 12:28:11,548] [INFO] [timer.py:197:stop] 0/1366, RunningAvgSamplesPerSec=5.862460106303159, CurrSamplesPerSec=5.326618605905622, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 675/5000 [2:05:47<13:18:08, 11.07s/it]
                                                       
{'loss': 0.0045, 'learning_rate': 9.602222222222222e-06, 'epoch': 16.46}

 14%|█▎        | 675/5000 [2:05:47<13:18:08, 11.07s/it][2022-12-20 12:28:22,639] [INFO] [timer.py:197:stop] 0/1368, RunningAvgSamplesPerSec=5.862435011779134, CurrSamplesPerSec=5.301754324980684, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 676/5000 [2:05:58<13:18:20, 11.08s/it][2022-12-20 12:28:33,892] [INFO] [timer.py:197:stop] 0/1370, RunningAvgSamplesPerSec=5.862267902829869, CurrSamplesPerSec=5.129299085198366, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 677/5000 [2:06:09<13:21:57, 11.13s/it][2022-12-20 12:28:44,911] [INFO] [timer.py:197:stop] 0/1372, RunningAvgSamplesPerSec=5.862284229165093, CurrSamplesPerSec=5.28853930648668, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 678/5000 [2:06:20<13:19:22, 11.10s/it][2022-12-20 12:28:56,007] [INFO] [timer.py:197:stop] 0/1374, RunningAvgSamplesPerSec=5.862254052668636, CurrSamplesPerSec=5.311328501216814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 679/5000 [2:06:31<13:19:08, 11.10s/it][2022-12-20 12:29:07,108] [INFO] [timer.py:197:stop] 0/1376, RunningAvgSamplesPerSec=5.862206919552749, CurrSamplesPerSec=5.326774619485562, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 680/5000 [2:06:42<13:19:02, 11.10s/it][2022-12-20 12:29:18,160] [INFO] [timer.py:197:stop] 0/1378, RunningAvgSamplesPerSec=5.862197809976487, CurrSamplesPerSec=5.311769500602897, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 681/5000 [2:06:53<13:17:53, 11.08s/it][2022-12-20 12:29:29,223] [INFO] [logging.py:68:log_dist] [Rank 0] step=690, skipped=3, lr=[9.586666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:29:29,225] [INFO] [timer.py:197:stop] 0/1380, RunningAvgSamplesPerSec=5.862191410659078, CurrSamplesPerSec=5.306218002829398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 682/5000 [2:07:04<13:17:16, 11.08s/it][2022-12-20 12:29:40,242] [INFO] [timer.py:197:stop] 0/1382, RunningAvgSamplesPerSec=5.862210262276334, CurrSamplesPerSec=5.32787289231465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 683/5000 [2:07:15<13:15:46, 11.06s/it][2022-12-20 12:29:51,243] [INFO] [timer.py:197:stop] 0/1384, RunningAvgSamplesPerSec=5.86224162252486, CurrSamplesPerSec=5.352946192288644, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 684/5000 [2:07:26<13:14:17, 11.04s/it][2022-12-20 12:30:02,264] [INFO] [timer.py:197:stop] 0/1386, RunningAvgSamplesPerSec=5.8622575018523895, CurrSamplesPerSec=5.3326300382972915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 685/5000 [2:07:37<13:13:39, 11.04s/it][2022-12-20 12:30:13,323] [INFO] [timer.py:197:stop] 0/1388, RunningAvgSamplesPerSec=5.862243883357019, CurrSamplesPerSec=5.306784464356799, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 686/5000 [2:07:48<13:13:58, 11.04s/it][2022-12-20 12:30:24,387] [INFO] [timer.py:197:stop] 0/1390, RunningAvgSamplesPerSec=5.862225716952207, CurrSamplesPerSec=5.2983602477773415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▎        | 687/5000 [2:07:59<13:14:15, 11.05s/it][2022-12-20 12:30:35,353] [INFO] [timer.py:197:stop] 0/1392, RunningAvgSamplesPerSec=5.862284281747808, CurrSamplesPerSec=5.351928255991902, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 688/5000 [2:08:10<13:12:16, 11.02s/it][2022-12-20 12:30:46,322] [INFO] [timer.py:197:stop] 0/1394, RunningAvgSamplesPerSec=5.862345869617665, CurrSamplesPerSec=5.358342655964066, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 689/5000 [2:08:21<13:10:54, 11.01s/it][2022-12-20 12:30:57,303] [INFO] [timer.py:197:stop] 0/1396, RunningAvgSamplesPerSec=5.8623931862851, CurrSamplesPerSec=5.337788379475375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 690/5000 [2:08:32<13:10:07, 11.00s/it][2022-12-20 12:31:08,339] [INFO] [timer.py:197:stop] 0/1398, RunningAvgSamplesPerSec=5.862396553242081, CurrSamplesPerSec=5.277941198445612, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 691/5000 [2:08:43<13:10:44, 11.01s/it][2022-12-20 12:31:19,366] [INFO] [logging.py:68:log_dist] [Rank 0] step=700, skipped=3, lr=[9.564444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:31:19,368] [INFO] [timer.py:197:stop] 0/1400, RunningAvgSamplesPerSec=5.862406448887152, CurrSamplesPerSec=5.304200913664271, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 692/5000 [2:08:54<13:10:57, 11.02s/it][2022-12-20 12:31:30,365] [INFO] [timer.py:197:stop] 0/1402, RunningAvgSamplesPerSec=5.862451744008267, CurrSamplesPerSec=5.32644336577709, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 693/5000 [2:09:05<13:10:21, 11.01s/it][2022-12-20 12:31:41,369] [INFO] [timer.py:197:stop] 0/1404, RunningAvgSamplesPerSec=5.862479622036666, CurrSamplesPerSec=5.327321161893251, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 694/5000 [2:09:16<13:10:01, 11.01s/it][2022-12-20 12:31:52,513] [INFO] [timer.py:197:stop] 0/1406, RunningAvgSamplesPerSec=5.862503506006748, CurrSamplesPerSec=5.33242537805458, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 695/5000 [2:09:28<13:12:46, 11.05s/it][2022-12-20 12:32:03,529] [INFO] [timer.py:197:stop] 0/1408, RunningAvgSamplesPerSec=5.862522818586606, CurrSamplesPerSec=5.313830429958, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 696/5000 [2:09:39<13:11:52, 11.04s/it][2022-12-20 12:32:14,564] [INFO] [timer.py:197:stop] 0/1410, RunningAvgSamplesPerSec=5.862527646952717, CurrSamplesPerSec=5.31581800663035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 697/5000 [2:09:50<13:11:36, 11.04s/it][2022-12-20 12:32:24,790] [INFO] [timer.py:197:stop] 0/1412, RunningAvgSamplesPerSec=5.863140415402685, CurrSamplesPerSec=5.314264269983449, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 698/5000 [2:10:05<14:40:30, 12.28s/it][2022-12-20 12:32:35,829] [INFO] [timer.py:197:stop] 0/1414, RunningAvgSamplesPerSec=5.8631519231603795, CurrSamplesPerSec=5.326942693120321, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 699/5000 [2:10:16<14:13:39, 11.91s/it][2022-12-20 12:32:47,138] [INFO] [timer.py:197:stop] 0/1416, RunningAvgSamplesPerSec=5.862945723258911, CurrSamplesPerSec=5.087825728518697, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 700/5000 [2:10:27<14:00:44, 11.73s/it]
                                                       
{'loss': 0.0044, 'learning_rate': 9.546666666666668e-06, 'epoch': 17.07}

 14%|█▍        | 700/5000 [2:10:27<14:00:44, 11.73s/it][2022-12-20 12:32:58,151] [INFO] [timer.py:197:stop] 0/1418, RunningAvgSamplesPerSec=5.862964586432788, CurrSamplesPerSec=5.345773187056398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 701/5000 [2:10:38<13:44:23, 11.51s/it][2022-12-20 12:33:09,145] [INFO] [logging.py:68:log_dist] [Rank 0] step=710, skipped=3, lr=[9.542222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:33:09,147] [INFO] [timer.py:197:stop] 0/1420, RunningAvgSamplesPerSec=5.863008252529106, CurrSamplesPerSec=5.3305229592870145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 702/5000 [2:10:49<13:34:01, 11.36s/it][2022-12-20 12:33:20,229] [INFO] [timer.py:197:stop] 0/1422, RunningAvgSamplesPerSec=5.862988228660555, CurrSamplesPerSec=5.302818000428436, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 703/5000 [2:11:00<13:27:01, 11.27s/it][2022-12-20 12:33:31,221] [INFO] [timer.py:197:stop] 0/1424, RunningAvgSamplesPerSec=5.8630247281163115, CurrSamplesPerSec=5.337364274080323, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 704/5000 [2:11:11<13:21:18, 11.19s/it][2022-12-20 12:33:42,286] [INFO] [timer.py:197:stop] 0/1426, RunningAvgSamplesPerSec=5.863005648204129, CurrSamplesPerSec=5.289533687120402, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 705/5000 [2:11:22<13:18:17, 11.15s/it][2022-12-20 12:33:53,326] [INFO] [timer.py:197:stop] 0/1428, RunningAvgSamplesPerSec=5.863017869777261, CurrSamplesPerSec=5.306560173089623, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 706/5000 [2:11:33<13:16:24, 11.13s/it][2022-12-20 12:34:04,348] [INFO] [timer.py:197:stop] 0/1430, RunningAvgSamplesPerSec=5.863029822754019, CurrSamplesPerSec=5.35103017247497, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 707/5000 [2:11:44<13:12:42, 11.08s/it][2022-12-20 12:34:15,585] [INFO] [timer.py:197:stop] 0/1432, RunningAvgSamplesPerSec=5.862881974422671, CurrSamplesPerSec=5.119417644370441, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 708/5000 [2:11:56<13:16:38, 11.14s/it][2022-12-20 12:34:26,638] [INFO] [timer.py:197:stop] 0/1434, RunningAvgSamplesPerSec=5.862870671407461, CurrSamplesPerSec=5.3029652896437245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 709/5000 [2:12:07<13:14:51, 11.11s/it][2022-12-20 12:34:37,695] [INFO] [timer.py:197:stop] 0/1436, RunningAvgSamplesPerSec=5.862857693347791, CurrSamplesPerSec=5.308013052349, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 710/5000 [2:12:18<13:14:21, 11.11s/it][2022-12-20 12:34:48,800] [INFO] [timer.py:197:stop] 0/1438, RunningAvgSamplesPerSec=5.862808100759116, CurrSamplesPerSec=5.303087024175011, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 711/5000 [2:12:29<13:12:11, 11.08s/it][2022-12-20 12:34:59,813] [INFO] [logging.py:68:log_dist] [Rank 0] step=720, skipped=3, lr=[9.52e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:34:59,815] [INFO] [timer.py:197:stop] 0/1440, RunningAvgSamplesPerSec=5.862838339145501, CurrSamplesPerSec=5.3180514887388695, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 712/5000 [2:12:40<13:10:52, 11.07s/it][2022-12-20 12:35:10,891] [INFO] [timer.py:197:stop] 0/1442, RunningAvgSamplesPerSec=5.862811043242973, CurrSamplesPerSec=5.264983263526601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 713/5000 [2:12:51<13:10:49, 11.07s/it][2022-12-20 12:35:21,917] [INFO] [timer.py:197:stop] 0/1444, RunningAvgSamplesPerSec=5.862821256297847, CurrSamplesPerSec=5.30550086819655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 714/5000 [2:13:02<13:10:14, 11.06s/it][2022-12-20 12:35:32,944] [INFO] [timer.py:197:stop] 0/1446, RunningAvgSamplesPerSec=5.862830060406905, CurrSamplesPerSec=5.324769336426582, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 715/5000 [2:13:13<13:09:57, 11.06s/it][2022-12-20 12:35:43,978] [INFO] [timer.py:197:stop] 0/1448, RunningAvgSamplesPerSec=5.86284803528612, CurrSamplesPerSec=5.362034031075395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 716/5000 [2:13:24<13:08:01, 11.04s/it][2022-12-20 12:35:54,976] [INFO] [timer.py:197:stop] 0/1450, RunningAvgSamplesPerSec=5.862879409807137, CurrSamplesPerSec=5.3304897219134695, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 717/5000 [2:13:35<13:07:54, 11.04s/it][2022-12-20 12:36:06,112] [INFO] [timer.py:197:stop] 0/1452, RunningAvgSamplesPerSec=5.86280841771356, CurrSamplesPerSec=5.24643774059284, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 718/5000 [2:13:46<13:10:04, 11.07s/it][2022-12-20 12:36:17,174] [INFO] [timer.py:197:stop] 0/1454, RunningAvgSamplesPerSec=5.862804571909612, CurrSamplesPerSec=5.321350378753188, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 719/5000 [2:13:57<13:09:31, 11.07s/it][2022-12-20 12:36:28,253] [INFO] [timer.py:197:stop] 0/1456, RunningAvgSamplesPerSec=5.8627761212437575, CurrSamplesPerSec=5.298971475993046, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 720/5000 [2:14:08<13:10:16, 11.08s/it][2022-12-20 12:36:39,361] [INFO] [timer.py:197:stop] 0/1458, RunningAvgSamplesPerSec=5.862726656670426, CurrSamplesPerSec=5.300774814045182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 721/5000 [2:14:19<13:10:17, 11.08s/it][2022-12-20 12:36:50,441] [INFO] [logging.py:68:log_dist] [Rank 0] step=730, skipped=3, lr=[9.49777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:36:50,443] [INFO] [timer.py:197:stop] 0/1460, RunningAvgSamplesPerSec=5.862696002260357, CurrSamplesPerSec=5.304338846363836, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 722/5000 [2:14:30<13:08:49, 11.06s/it][2022-12-20 12:37:01,811] [INFO] [timer.py:197:stop] 0/1462, RunningAvgSamplesPerSec=5.862463880905035, CurrSamplesPerSec=5.026119374340526, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 723/5000 [2:14:42<13:16:32, 11.17s/it][2022-12-20 12:37:12,874] [INFO] [timer.py:197:stop] 0/1464, RunningAvgSamplesPerSec=5.862461870571878, CurrSamplesPerSec=5.325794929616139, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 724/5000 [2:14:53<13:13:27, 11.13s/it][2022-12-20 12:37:23,904] [INFO] [timer.py:197:stop] 0/1466, RunningAvgSamplesPerSec=5.862480115386148, CurrSamplesPerSec=5.346091730481357, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 14%|█▍        | 725/5000 [2:15:04<13:10:48, 11.10s/it]
                                                       
{'loss': 0.0031, 'learning_rate': 9.491111111111112e-06, 'epoch': 17.67}

 14%|█▍        | 725/5000 [2:15:04<13:10:48, 11.10s/it][2022-12-20 12:37:35,204] [INFO] [timer.py:197:stop] 0/1468, RunningAvgSamplesPerSec=5.862291782943352, CurrSamplesPerSec=5.096142547389954, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 726/5000 [2:15:15<13:14:49, 11.16s/it][2022-12-20 12:37:46,223] [INFO] [timer.py:197:stop] 0/1470, RunningAvgSamplesPerSec=5.862307777963867, CurrSamplesPerSec=5.329184265640708, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 727/5000 [2:15:26<13:12:15, 11.12s/it][2022-12-20 12:37:57,282] [INFO] [timer.py:197:stop] 0/1472, RunningAvgSamplesPerSec=5.862306671247865, CurrSamplesPerSec=5.31854945460962, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 728/5000 [2:15:37<13:09:56, 11.09s/it][2022-12-20 12:38:08,319] [INFO] [timer.py:197:stop] 0/1474, RunningAvgSamplesPerSec=5.862310491487235, CurrSamplesPerSec=5.3087289765315555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 729/5000 [2:15:48<13:08:59, 11.08s/it][2022-12-20 12:38:19,308] [INFO] [timer.py:197:stop] 0/1476, RunningAvgSamplesPerSec=5.862348152548847, CurrSamplesPerSec=5.36991821078434, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 730/5000 [2:15:59<13:06:20, 11.05s/it][2022-12-20 12:38:30,406] [INFO] [timer.py:197:stop] 0/1478, RunningAvgSamplesPerSec=5.862306654395147, CurrSamplesPerSec=5.255676340088974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 731/5000 [2:16:10<13:07:38, 11.07s/it][2022-12-20 12:38:41,447] [INFO] [logging.py:68:log_dist] [Rank 0] step=740, skipped=3, lr=[9.475555555555557e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:38:41,449] [INFO] [timer.py:197:stop] 0/1480, RunningAvgSamplesPerSec=5.8623162662281105, CurrSamplesPerSec=5.321017478859083, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 732/5000 [2:16:21<13:06:33, 11.06s/it][2022-12-20 12:38:52,519] [INFO] [timer.py:197:stop] 0/1482, RunningAvgSamplesPerSec=5.862295008432646, CurrSamplesPerSec=5.285097815538616, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 733/5000 [2:16:33<13:06:00, 11.05s/it][2022-12-20 12:39:03,530] [INFO] [timer.py:197:stop] 0/1484, RunningAvgSamplesPerSec=5.862317977910438, CurrSamplesPerSec=5.311972578327592, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 734/5000 [2:16:44<13:05:07, 11.04s/it][2022-12-20 12:39:14,554] [INFO] [timer.py:197:stop] 0/1486, RunningAvgSamplesPerSec=5.862330312042108, CurrSamplesPerSec=5.3071772820181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 735/5000 [2:16:55<13:05:08, 11.05s/it][2022-12-20 12:39:25,628] [INFO] [timer.py:197:stop] 0/1488, RunningAvgSamplesPerSec=5.8623059350756925, CurrSamplesPerSec=5.2870731158810775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 736/5000 [2:17:06<13:05:18, 11.05s/it][2022-12-20 12:39:36,618] [INFO] [timer.py:197:stop] 0/1490, RunningAvgSamplesPerSec=5.862341322980358, CurrSamplesPerSec=5.35014667909395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 737/5000 [2:17:17<13:04:12, 11.04s/it][2022-12-20 12:39:47,619] [INFO] [timer.py:197:stop] 0/1492, RunningAvgSamplesPerSec=5.862369119676411, CurrSamplesPerSec=5.3558749439892175, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 738/5000 [2:17:28<13:03:09, 11.03s/it][2022-12-20 12:39:57,831] [INFO] [timer.py:197:stop] 0/1494, RunningAvgSamplesPerSec=5.862956465063401, CurrSamplesPerSec=6.152339209660849, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 12:40:08,820] [INFO] [timer.py:197:stop] 0/1496, RunningAvgSamplesPerSec=5.862990577880877, CurrSamplesPerSec=5.312042166502927, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 739/5000 [2:17:44<14:54:08, 12.59s/it][2022-12-20 12:40:19,792] [INFO] [timer.py:197:stop] 0/1498, RunningAvgSamplesPerSec=5.8630518610997004, CurrSamplesPerSec=5.346287432064414, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 740/5000 [2:17:55<14:19:26, 12.10s/it][2022-12-20 12:40:30,923] [INFO] [logging.py:68:log_dist] [Rank 0] step=750, skipped=3, lr=[9.453333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:40:30,925] [INFO] [timer.py:197:stop] 0/1500, RunningAvgSamplesPerSec=5.862994696221895, CurrSamplesPerSec=5.28199315934495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 741/5000 [2:18:06<13:58:32, 11.81s/it][2022-12-20 12:40:41,979] [INFO] [timer.py:197:stop] 0/1502, RunningAvgSamplesPerSec=5.862984952469361, CurrSamplesPerSec=5.313633100369348, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 742/5000 [2:18:17<13:42:10, 11.59s/it][2022-12-20 12:40:53,101] [INFO] [timer.py:197:stop] 0/1504, RunningAvgSamplesPerSec=5.862925618699052, CurrSamplesPerSec=5.230900624573559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 743/5000 [2:18:28<13:32:07, 11.45s/it][2022-12-20 12:41:04,131] [INFO] [timer.py:197:stop] 0/1506, RunningAvgSamplesPerSec=5.862942259627045, CurrSamplesPerSec=5.339198515532887, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 744/5000 [2:18:39<13:23:03, 11.32s/it][2022-12-20 12:41:15,180] [INFO] [timer.py:197:stop] 0/1508, RunningAvgSamplesPerSec=5.862935243895618, CurrSamplesPerSec=5.321536044441882, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 745/5000 [2:18:50<13:17:04, 11.24s/it][2022-12-20 12:41:26,250] [INFO] [timer.py:197:stop] 0/1510, RunningAvgSamplesPerSec=5.86291325734134, CurrSamplesPerSec=5.244780618351424, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 746/5000 [2:19:01<13:13:16, 11.19s/it][2022-12-20 12:41:37,268] [INFO] [timer.py:197:stop] 0/1512, RunningAvgSamplesPerSec=5.862927938680188, CurrSamplesPerSec=5.334538626139491, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 747/5000 [2:19:12<13:09:29, 11.14s/it][2022-12-20 12:41:48,327] [INFO] [timer.py:197:stop] 0/1514, RunningAvgSamplesPerSec=5.86292599481629, CurrSamplesPerSec=5.3216461840748215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 748/5000 [2:19:23<13:07:36, 11.11s/it][2022-12-20 12:41:59,468] [INFO] [timer.py:197:stop] 0/1516, RunningAvgSamplesPerSec=5.862854016510408, CurrSamplesPerSec=5.2602756906129535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▍        | 749/5000 [2:19:35<13:07:58, 11.12s/it][2022-12-20 12:42:10,521] [INFO] [timer.py:197:stop] 0/1518, RunningAvgSamplesPerSec=5.86284397174031, CurrSamplesPerSec=5.304293986005706, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 750/5000 [2:19:46<13:06:21, 11.10s/it]
                                                       
{'loss': 0.0032, 'learning_rate': 9.433333333333335e-06, 'epoch': 18.29}

 15%|█▌        | 750/5000 [2:19:46<13:06:21, 11.10s/it][2022-12-20 12:42:21,631] [INFO] [logging.py:68:log_dist] [Rank 0] step=760, skipped=3, lr=[9.431111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:42:21,633] [INFO] [timer.py:197:stop] 0/1520, RunningAvgSamplesPerSec=5.8627931968038824, CurrSamplesPerSec=5.230963619672965, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 751/5000 [2:19:57<13:06:23, 11.10s/it][2022-12-20 12:42:32,629] [INFO] [timer.py:197:stop] 0/1522, RunningAvgSamplesPerSec=5.86282352775567, CurrSamplesPerSec=5.344009961001843, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 752/5000 [2:20:08<13:03:54, 11.07s/it][2022-12-20 12:42:43,653] [INFO] [timer.py:197:stop] 0/1524, RunningAvgSamplesPerSec=5.8628349814828065, CurrSamplesPerSec=5.320646232798189, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 753/5000 [2:20:19<13:02:41, 11.06s/it][2022-12-20 12:42:54,742] [INFO] [timer.py:197:stop] 0/1526, RunningAvgSamplesPerSec=5.862811325107653, CurrSamplesPerSec=5.302987708492099, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 754/5000 [2:20:30<13:03:10, 11.07s/it][2022-12-20 12:43:05,764] [INFO] [timer.py:197:stop] 0/1528, RunningAvgSamplesPerSec=5.862823558052423, CurrSamplesPerSec=5.339983426753459, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 755/5000 [2:20:41<13:02:02, 11.05s/it][2022-12-20 12:43:16,805] [INFO] [timer.py:197:stop] 0/1530, RunningAvgSamplesPerSec=5.862822188730468, CurrSamplesPerSec=5.309198106061236, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 756/5000 [2:20:52<13:01:35, 11.05s/it][2022-12-20 12:43:27,775] [INFO] [timer.py:197:stop] 0/1532, RunningAvgSamplesPerSec=5.862871263520785, CurrSamplesPerSec=5.325916023858222, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 757/5000 [2:21:03<12:59:42, 11.03s/it][2022-12-20 12:43:38,827] [INFO] [timer.py:197:stop] 0/1534, RunningAvgSamplesPerSec=5.862861424451835, CurrSamplesPerSec=5.307026401262352, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 758/5000 [2:21:14<13:00:05, 11.03s/it][2022-12-20 12:43:50,028] [INFO] [timer.py:197:stop] 0/1536, RunningAvgSamplesPerSec=5.862748640204478, CurrSamplesPerSec=5.186568654239636, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 759/5000 [2:21:25<13:03:26, 11.08s/it][2022-12-20 12:44:01,137] [INFO] [timer.py:197:stop] 0/1538, RunningAvgSamplesPerSec=5.862700574871624, CurrSamplesPerSec=5.2823322123700365, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 760/5000 [2:21:36<13:03:48, 11.09s/it][2022-12-20 12:44:12,126] [INFO] [logging.py:68:log_dist] [Rank 0] step=770, skipped=3, lr=[9.40888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:44:12,128] [INFO] [timer.py:197:stop] 0/1540, RunningAvgSamplesPerSec=5.862746253646926, CurrSamplesPerSec=5.334013707796813, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 761/5000 [2:21:47<13:01:28, 11.06s/it][2022-12-20 12:44:23,185] [INFO] [timer.py:197:stop] 0/1542, RunningAvgSamplesPerSec=5.862734016513703, CurrSamplesPerSec=5.335871738451829, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 762/5000 [2:21:58<13:01:12, 11.06s/it][2022-12-20 12:44:34,234] [INFO] [timer.py:197:stop] 0/1544, RunningAvgSamplesPerSec=5.862727010988443, CurrSamplesPerSec=5.305943836926476, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 763/5000 [2:22:09<13:00:46, 11.06s/it][2022-12-20 12:44:45,387] [INFO] [timer.py:197:stop] 0/1546, RunningAvgSamplesPerSec=5.862648055955911, CurrSamplesPerSec=5.241167407602945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 764/5000 [2:22:20<13:02:38, 11.09s/it][2022-12-20 12:44:56,399] [INFO] [timer.py:197:stop] 0/1548, RunningAvgSamplesPerSec=5.862667271317212, CurrSamplesPerSec=5.337734884990586, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 765/5000 [2:22:31<13:00:53, 11.06s/it][2022-12-20 12:45:07,422] [INFO] [timer.py:197:stop] 0/1550, RunningAvgSamplesPerSec=5.862677685292013, CurrSamplesPerSec=5.3364108137345205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 766/5000 [2:22:43<12:59:51, 11.05s/it][2022-12-20 12:45:18,511] [INFO] [timer.py:197:stop] 0/1552, RunningAvgSamplesPerSec=5.862655405450703, CurrSamplesPerSec=5.3057238114890515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 767/5000 [2:22:54<13:00:28, 11.06s/it][2022-12-20 12:45:29,520] [INFO] [timer.py:197:stop] 0/1554, RunningAvgSamplesPerSec=5.862676887396045, CurrSamplesPerSec=5.322840500101664, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 768/5000 [2:23:05<12:59:08, 11.05s/it][2022-12-20 12:45:40,953] [INFO] [timer.py:197:stop] 0/1556, RunningAvgSamplesPerSec=5.86240478258947, CurrSamplesPerSec=4.986368726772981, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 769/5000 [2:23:16<13:07:09, 11.16s/it][2022-12-20 12:45:51,966] [INFO] [timer.py:197:stop] 0/1558, RunningAvgSamplesPerSec=5.862424402120004, CurrSamplesPerSec=5.320771522552136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 770/5000 [2:23:27<13:03:47, 11.12s/it][2022-12-20 12:46:02,965] [INFO] [logging.py:68:log_dist] [Rank 0] step=780, skipped=3, lr=[9.386666666666668e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:46:02,966] [INFO] [timer.py:197:stop] 0/1560, RunningAvgSamplesPerSec=5.862450773369622, CurrSamplesPerSec=5.357572227459336, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 771/5000 [2:23:38<13:01:07, 11.08s/it][2022-12-20 12:46:14,091] [INFO] [timer.py:197:stop] 0/1562, RunningAvgSamplesPerSec=5.862392704768418, CurrSamplesPerSec=5.249518795742663, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 772/5000 [2:23:49<13:01:49, 11.09s/it][2022-12-20 12:46:25,122] [INFO] [timer.py:197:stop] 0/1564, RunningAvgSamplesPerSec=5.862398153805768, CurrSamplesPerSec=5.298704124741076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 773/5000 [2:24:00<13:00:17, 11.08s/it][2022-12-20 12:46:36,074] [INFO] [timer.py:197:stop] 0/1566, RunningAvgSamplesPerSec=5.86245829374021, CurrSamplesPerSec=5.352165362777042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 15%|█▌        | 774/5000 [2:24:11<12:57:29, 11.04s/it][2022-12-20 12:46:47,148] [INFO] [timer.py:197:stop] 0/1568, RunningAvgSamplesPerSec=5.862435245177273, CurrSamplesPerSec=5.304353520477455, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 775/5000 [2:24:22<12:58:02, 11.05s/it]
                                                       
{'loss': 0.0024, 'learning_rate': 9.377777777777779e-06, 'epoch': 18.89}

 16%|█▌        | 775/5000 [2:24:22<12:58:02, 11.05s/it][2022-12-20 12:46:58,185] [INFO] [timer.py:197:stop] 0/1570, RunningAvgSamplesPerSec=5.862449227356594, CurrSamplesPerSec=5.326821340777515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 776/5000 [2:24:33<12:57:37, 11.05s/it][2022-12-20 12:47:09,222] [INFO] [timer.py:197:stop] 0/1572, RunningAvgSamplesPerSec=5.862451298943747, CurrSamplesPerSec=5.31278483423233, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 777/5000 [2:24:44<12:57:15, 11.04s/it][2022-12-20 12:47:20,214] [INFO] [timer.py:197:stop] 0/1574, RunningAvgSamplesPerSec=5.862484081238911, CurrSamplesPerSec=5.334855619995085, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 778/5000 [2:24:55<12:55:59, 11.03s/it][2022-12-20 12:47:31,270] [INFO] [timer.py:197:stop] 0/1576, RunningAvgSamplesPerSec=5.862493911403284, CurrSamplesPerSec=5.316813405726989, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 779/5000 [2:25:06<12:56:24, 11.04s/it][2022-12-20 12:47:41,390] [INFO] [timer.py:197:stop] 0/1578, RunningAvgSamplesPerSec=5.863126374331837, CurrSamplesPerSec=5.334717791950672, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 780/5000 [2:25:21<14:21:40, 12.25s/it][2022-12-20 12:47:52,424] [INFO] [logging.py:68:log_dist] [Rank 0] step=790, skipped=3, lr=[9.364444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:47:52,426] [INFO] [timer.py:197:stop] 0/1580, RunningAvgSamplesPerSec=5.863127220966555, CurrSamplesPerSec=5.327345690216709, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 781/5000 [2:25:32<13:53:57, 11.86s/it][2022-12-20 12:48:03,403] [INFO] [timer.py:197:stop] 0/1582, RunningAvgSamplesPerSec=5.86316803018582, CurrSamplesPerSec=5.301090320267187, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 782/5000 [2:25:43<13:37:23, 11.63s/it][2022-12-20 12:48:14,441] [INFO] [timer.py:197:stop] 0/1584, RunningAvgSamplesPerSec=5.863184205863818, CurrSamplesPerSec=5.342129249078624, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 783/5000 [2:25:54<13:23:54, 11.44s/it][2022-12-20 12:48:25,446] [INFO] [timer.py:197:stop] 0/1586, RunningAvgSamplesPerSec=5.863206789626215, CurrSamplesPerSec=5.335937075109575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 784/5000 [2:26:06<13:17:32, 11.35s/it][2022-12-20 12:48:36,633] [INFO] [timer.py:197:stop] 0/1588, RunningAvgSamplesPerSec=5.863126000322619, CurrSamplesPerSec=5.312889143951293, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 785/5000 [2:26:17<13:10:28, 11.25s/it][2022-12-20 12:48:47,609] [INFO] [timer.py:197:stop] 0/1590, RunningAvgSamplesPerSec=5.863168834968209, CurrSamplesPerSec=5.341165371120974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 786/5000 [2:26:28<13:04:56, 11.18s/it][2022-12-20 12:48:58,646] [INFO] [timer.py:197:stop] 0/1592, RunningAvgSamplesPerSec=5.863171862200918, CurrSamplesPerSec=5.309412118821607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 787/5000 [2:26:39<13:02:05, 11.14s/it][2022-12-20 12:49:09,690] [INFO] [timer.py:197:stop] 0/1594, RunningAvgSamplesPerSec=5.8631872292179805, CurrSamplesPerSec=5.311813646613485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 788/5000 [2:26:50<12:59:10, 11.10s/it][2022-12-20 12:49:20,657] [INFO] [timer.py:197:stop] 0/1596, RunningAvgSamplesPerSec=5.863235012809538, CurrSamplesPerSec=5.3482384165662875, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 789/5000 [2:27:01<12:56:34, 11.07s/it][2022-12-20 12:49:31,855] [INFO] [timer.py:197:stop] 0/1598, RunningAvgSamplesPerSec=5.8631270568480724, CurrSamplesPerSec=5.163991159429903, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 790/5000 [2:27:12<12:58:26, 11.09s/it][2022-12-20 12:49:42,857] [INFO] [logging.py:68:log_dist] [Rank 0] step=800, skipped=3, lr=[9.342222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:49:42,859] [INFO] [timer.py:197:stop] 0/1600, RunningAvgSamplesPerSec=5.863149703132527, CurrSamplesPerSec=5.299126711086855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 791/5000 [2:27:23<12:57:07, 11.08s/it][2022-12-20 12:49:53,824] [INFO] [timer.py:197:stop] 0/1602, RunningAvgSamplesPerSec=5.86319839355764, CurrSamplesPerSec=5.36609068239062, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 792/5000 [2:27:34<12:57:02, 11.08s/it][2022-12-20 12:50:04,926] [INFO] [timer.py:197:stop] 0/1604, RunningAvgSamplesPerSec=5.86315602905558, CurrSamplesPerSec=5.349912949838555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 793/5000 [2:27:45<12:55:26, 11.06s/it][2022-12-20 12:50:15,985] [INFO] [timer.py:197:stop] 0/1606, RunningAvgSamplesPerSec=5.863154135041845, CurrSamplesPerSec=5.32364700346622, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 794/5000 [2:27:56<12:55:04, 11.06s/it][2022-12-20 12:50:27,380] [INFO] [timer.py:197:stop] 0/1608, RunningAvgSamplesPerSec=5.862915958799281, CurrSamplesPerSec=5.0218568965065025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 795/5000 [2:28:07<13:01:53, 11.16s/it][2022-12-20 12:50:38,455] [INFO] [timer.py:197:stop] 0/1610, RunningAvgSamplesPerSec=5.862891264768254, CurrSamplesPerSec=5.282053857239669, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 796/5000 [2:28:18<12:59:42, 11.13s/it][2022-12-20 12:50:49,437] [INFO] [timer.py:197:stop] 0/1612, RunningAvgSamplesPerSec=5.862937024147695, CurrSamplesPerSec=5.352586060269257, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 797/5000 [2:28:30<13:02:59, 11.18s/it][2022-12-20 12:51:00,709] [INFO] [timer.py:197:stop] 0/1614, RunningAvgSamplesPerSec=5.862781838661974, CurrSamplesPerSec=5.37184670822534, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 798/5000 [2:28:41<12:58:01, 11.11s/it][2022-12-20 12:51:11,717] [INFO] [timer.py:197:stop] 0/1616, RunningAvgSamplesPerSec=5.862802098911779, CurrSamplesPerSec=5.319331889799745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 799/5000 [2:28:52<12:56:11, 11.09s/it][2022-12-20 12:51:22,803] [INFO] [timer.py:197:stop] 0/1618, RunningAvgSamplesPerSec=5.862771542746577, CurrSamplesPerSec=5.271752730737156, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 800/5000 [2:29:03<12:55:00, 11.07s/it]
                                                       
{'loss': 0.0023, 'learning_rate': 9.322222222222223e-06, 'epoch': 19.51}

 16%|█▌        | 800/5000 [2:29:03<12:55:00, 11.07s/it][2022-12-20 12:51:33,760] [INFO] [logging.py:68:log_dist] [Rank 0] step=810, skipped=3, lr=[9.32e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:51:33,762] [INFO] [timer.py:197:stop] 0/1620, RunningAvgSamplesPerSec=5.862824607957821, CurrSamplesPerSec=5.341264846615713, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 801/5000 [2:29:14<12:52:58, 11.05s/it][2022-12-20 12:51:44,793] [INFO] [timer.py:197:stop] 0/1622, RunningAvgSamplesPerSec=5.862829325474067, CurrSamplesPerSec=5.298600580447793, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 802/5000 [2:29:25<12:52:54, 11.05s/it][2022-12-20 12:51:55,844] [INFO] [timer.py:197:stop] 0/1624, RunningAvgSamplesPerSec=5.862833454276366, CurrSamplesPerSec=5.314378948557945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 803/5000 [2:29:36<12:52:19, 11.04s/it][2022-12-20 12:52:06,842] [INFO] [timer.py:197:stop] 0/1626, RunningAvgSamplesPerSec=5.862860462375713, CurrSamplesPerSec=5.32550288902114, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 804/5000 [2:29:47<12:50:45, 11.02s/it][2022-12-20 12:52:17,944] [INFO] [timer.py:197:stop] 0/1628, RunningAvgSamplesPerSec=5.862820706339032, CurrSamplesPerSec=5.217581455728313, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 805/5000 [2:29:58<12:56:37, 11.11s/it][2022-12-20 12:52:29,152] [INFO] [timer.py:197:stop] 0/1630, RunningAvgSamplesPerSec=5.862809928323338, CurrSamplesPerSec=5.302270818283576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 806/5000 [2:30:09<12:55:39, 11.10s/it][2022-12-20 12:52:40,211] [INFO] [timer.py:197:stop] 0/1632, RunningAvgSamplesPerSec=5.862796703679118, CurrSamplesPerSec=5.312893560380581, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 807/5000 [2:30:20<12:54:05, 11.08s/it][2022-12-20 12:52:51,213] [INFO] [timer.py:197:stop] 0/1634, RunningAvgSamplesPerSec=5.862820934131541, CurrSamplesPerSec=5.338218284800875, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 808/5000 [2:30:31<12:52:39, 11.06s/it][2022-12-20 12:53:02,272] [INFO] [timer.py:197:stop] 0/1636, RunningAvgSamplesPerSec=5.862807453751224, CurrSamplesPerSec=5.301604171234681, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 809/5000 [2:30:42<12:51:20, 11.04s/it][2022-12-20 12:53:13,236] [INFO] [timer.py:197:stop] 0/1638, RunningAvgSamplesPerSec=5.862857037655219, CurrSamplesPerSec=5.338730015471123, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 810/5000 [2:30:53<12:50:03, 11.03s/it][2022-12-20 12:53:24,256] [INFO] [logging.py:68:log_dist] [Rank 0] step=820, skipped=3, lr=[9.297777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:53:24,258] [INFO] [timer.py:197:stop] 0/1640, RunningAvgSamplesPerSec=5.862868065820694, CurrSamplesPerSec=5.310558504377071, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 811/5000 [2:31:04<12:50:09, 11.03s/it][2022-12-20 12:53:35,300] [INFO] [timer.py:197:stop] 0/1642, RunningAvgSamplesPerSec=5.862865560465374, CurrSamplesPerSec=5.308891503517605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▌        | 812/5000 [2:31:15<12:49:44, 11.03s/it][2022-12-20 12:53:46,338] [INFO] [timer.py:197:stop] 0/1644, RunningAvgSamplesPerSec=5.8628655125708775, CurrSamplesPerSec=5.292708994464881, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 813/5000 [2:31:26<12:50:45, 11.05s/it][2022-12-20 12:53:57,349] [INFO] [timer.py:197:stop] 0/1646, RunningAvgSamplesPerSec=5.8628835085430495, CurrSamplesPerSec=5.358553589237677, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 814/5000 [2:31:37<12:49:20, 11.03s/it][2022-12-20 12:54:08,451] [INFO] [timer.py:197:stop] 0/1648, RunningAvgSamplesPerSec=5.862842391383693, CurrSamplesPerSec=5.256879316034818, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 815/5000 [2:31:48<12:50:18, 11.04s/it][2022-12-20 12:54:19,430] [INFO] [timer.py:197:stop] 0/1650, RunningAvgSamplesPerSec=5.862881820185489, CurrSamplesPerSec=5.3478388579863765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 816/5000 [2:31:59<12:49:22, 11.03s/it][2022-12-20 12:54:30,471] [INFO] [timer.py:197:stop] 0/1652, RunningAvgSamplesPerSec=5.862881285933096, CurrSamplesPerSec=5.319494223344462, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 817/5000 [2:32:11<12:51:34, 11.07s/it][2022-12-20 12:54:41,610] [INFO] [timer.py:197:stop] 0/1654, RunningAvgSamplesPerSec=5.862816053161566, CurrSamplesPerSec=5.325219754336654, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 818/5000 [2:32:22<12:50:35, 11.06s/it][2022-12-20 12:54:52,685] [INFO] [timer.py:197:stop] 0/1656, RunningAvgSamplesPerSec=5.862804879186885, CurrSamplesPerSec=5.300856461024815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 819/5000 [2:32:33<12:51:01, 11.06s/it][2022-12-20 12:55:04,135] [INFO] [timer.py:197:stop] 0/1658, RunningAvgSamplesPerSec=5.862538504273531, CurrSamplesPerSec=4.9852911734932235, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 820/5000 [2:32:44<12:57:52, 11.17s/it][2022-12-20 12:55:14,242] [INFO] [logging.py:68:log_dist] [Rank 0] step=830, skipped=3, lr=[9.275555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:55:14,244] [INFO] [timer.py:197:stop] 0/1660, RunningAvgSamplesPerSec=5.863135110375824, CurrSamplesPerSec=6.2291443121593915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 12:55:25,249] [INFO] [timer.py:197:stop] 0/1662, RunningAvgSamplesPerSec=5.863162502477956, CurrSamplesPerSec=5.360764252946563, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 821/5000 [2:33:00<14:42:47, 12.67s/it][2022-12-20 12:55:36,266] [INFO] [timer.py:197:stop] 0/1664, RunningAvgSamplesPerSec=5.863177310999006, CurrSamplesPerSec=5.324445513600693, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 822/5000 [2:33:11<14:07:56, 12.18s/it][2022-12-20 12:55:47,300] [INFO] [timer.py:197:stop] 0/1666, RunningAvgSamplesPerSec=5.863180134167597, CurrSamplesPerSec=5.318373691301102, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 823/5000 [2:33:22<13:43:52, 11.83s/it][2022-12-20 12:55:58,449] [INFO] [timer.py:197:stop] 0/1668, RunningAvgSamplesPerSec=5.863109284869318, CurrSamplesPerSec=5.220202705356924, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 824/5000 [2:33:34<13:29:22, 11.63s/it][2022-12-20 12:56:09,481] [INFO] [timer.py:197:stop] 0/1670, RunningAvgSamplesPerSec=5.863124526543233, CurrSamplesPerSec=5.315903065310432, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 16%|█▋        | 825/5000 [2:33:45<13:16:41, 11.45s/it]
                                                       
{'loss': 0.0025, 'learning_rate': 9.264444444444445e-06, 'epoch': 20.12}

 16%|█▋        | 825/5000 [2:33:45<13:16:41, 11.45s/it][2022-12-20 12:56:20,554] [INFO] [timer.py:197:stop] 0/1672, RunningAvgSamplesPerSec=5.863103279714837, CurrSamplesPerSec=5.309057400497197, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 826/5000 [2:33:56<13:08:39, 11.34s/it][2022-12-20 12:56:31,495] [INFO] [timer.py:197:stop] 0/1674, RunningAvgSamplesPerSec=5.863167275823387, CurrSamplesPerSec=5.353551502559773, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 827/5000 [2:34:07<13:00:13, 11.22s/it][2022-12-20 12:56:42,506] [INFO] [timer.py:197:stop] 0/1676, RunningAvgSamplesPerSec=5.863185570084754, CurrSamplesPerSec=5.3071703568442485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 828/5000 [2:34:18<12:55:42, 11.16s/it][2022-12-20 12:56:53,593] [INFO] [timer.py:197:stop] 0/1678, RunningAvgSamplesPerSec=5.863154472215378, CurrSamplesPerSec=5.2397637733027915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 829/5000 [2:34:29<12:54:06, 11.14s/it][2022-12-20 12:57:04,659] [INFO] [logging.py:68:log_dist] [Rank 0] step=840, skipped=3, lr=[9.253333333333333e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:57:04,661] [INFO] [timer.py:197:stop] 0/1680, RunningAvgSamplesPerSec=5.863146735464434, CurrSamplesPerSec=5.289621033736822, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 830/5000 [2:34:40<12:52:29, 11.11s/it][2022-12-20 12:57:15,719] [INFO] [timer.py:197:stop] 0/1682, RunningAvgSamplesPerSec=5.8631355538401, CurrSamplesPerSec=5.3164536966159615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 831/5000 [2:34:51<12:51:06, 11.10s/it][2022-12-20 12:57:26,839] [INFO] [timer.py:197:stop] 0/1684, RunningAvgSamplesPerSec=5.863084404061456, CurrSamplesPerSec=5.251763892005598, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 832/5000 [2:35:02<12:51:23, 11.10s/it][2022-12-20 12:57:37,798] [INFO] [timer.py:197:stop] 0/1686, RunningAvgSamplesPerSec=5.8631358499708, CurrSamplesPerSec=5.3491381213725715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 833/5000 [2:35:13<12:48:10, 11.06s/it][2022-12-20 12:57:48,836] [INFO] [timer.py:197:stop] 0/1688, RunningAvgSamplesPerSec=5.863143859732603, CurrSamplesPerSec=5.334948286853204, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 834/5000 [2:35:24<12:47:31, 11.05s/it][2022-12-20 12:57:59,933] [INFO] [timer.py:197:stop] 0/1690, RunningAvgSamplesPerSec=5.863106923809662, CurrSamplesPerSec=5.338588164942813, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 835/5000 [2:35:35<12:48:13, 11.07s/it][2022-12-20 12:58:11,012] [INFO] [timer.py:197:stop] 0/1692, RunningAvgSamplesPerSec=5.8630816870390285, CurrSamplesPerSec=5.286192917248058, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 836/5000 [2:35:46<12:48:18, 11.07s/it][2022-12-20 12:58:22,052] [INFO] [timer.py:197:stop] 0/1694, RunningAvgSamplesPerSec=5.863081725208077, CurrSamplesPerSec=5.272264016133982, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 837/5000 [2:35:57<12:47:29, 11.06s/it][2022-12-20 12:58:33,058] [INFO] [timer.py:197:stop] 0/1696, RunningAvgSamplesPerSec=5.863114340793275, CurrSamplesPerSec=5.318639237472861, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 838/5000 [2:36:08<12:46:07, 11.04s/it][2022-12-20 12:58:44,124] [INFO] [timer.py:197:stop] 0/1698, RunningAvgSamplesPerSec=5.863097258922048, CurrSamplesPerSec=5.305821971202759, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 839/5000 [2:36:19<12:46:23, 11.05s/it][2022-12-20 12:58:55,122] [INFO] [logging.py:68:log_dist] [Rank 0] step=850, skipped=3, lr=[9.231111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 12:58:55,124] [INFO] [timer.py:197:stop] 0/1700, RunningAvgSamplesPerSec=5.8631323763817855, CurrSamplesPerSec=5.352242837864453, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 840/5000 [2:36:30<12:45:08, 11.04s/it][2022-12-20 12:59:06,205] [INFO] [timer.py:197:stop] 0/1702, RunningAvgSamplesPerSec=5.863105956502683, CurrSamplesPerSec=5.283776845579854, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 841/5000 [2:36:41<12:45:54, 11.05s/it][2022-12-20 12:59:17,500] [INFO] [timer.py:197:stop] 0/1704, RunningAvgSamplesPerSec=5.8629512629977665, CurrSamplesPerSec=5.103308499498482, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 842/5000 [2:36:53<12:51:02, 11.13s/it][2022-12-20 12:59:28,626] [INFO] [timer.py:197:stop] 0/1706, RunningAvgSamplesPerSec=5.862977349225501, CurrSamplesPerSec=5.353332422084103, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 843/5000 [2:37:04<12:50:37, 11.12s/it][2022-12-20 12:59:39,630] [INFO] [timer.py:197:stop] 0/1708, RunningAvgSamplesPerSec=5.8629990940262475, CurrSamplesPerSec=5.345260743978744, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 844/5000 [2:37:15<12:47:58, 11.09s/it][2022-12-20 12:59:50,784] [INFO] [timer.py:197:stop] 0/1710, RunningAvgSamplesPerSec=5.862926810048994, CurrSamplesPerSec=5.221606644212056, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 845/5000 [2:37:26<12:49:12, 11.11s/it][2022-12-20 13:00:01,783] [INFO] [timer.py:197:stop] 0/1712, RunningAvgSamplesPerSec=5.8629605948550365, CurrSamplesPerSec=5.321311348409897, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 846/5000 [2:37:37<12:46:44, 11.07s/it][2022-12-20 13:00:12,776] [INFO] [timer.py:197:stop] 0/1714, RunningAvgSamplesPerSec=5.862998588739233, CurrSamplesPerSec=5.339276040470345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 847/5000 [2:37:48<12:44:51, 11.05s/it][2022-12-20 13:00:23,839] [INFO] [timer.py:197:stop] 0/1716, RunningAvgSamplesPerSec=5.862994769935858, CurrSamplesPerSec=5.349537874129724, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 848/5000 [2:37:59<12:44:56, 11.05s/it][2022-12-20 13:00:34,842] [INFO] [timer.py:197:stop] 0/1718, RunningAvgSamplesPerSec=5.863017836316093, CurrSamplesPerSec=5.321035831570626, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 849/5000 [2:38:10<12:43:40, 11.04s/it][2022-12-20 13:00:45,892] [INFO] [logging.py:68:log_dist] [Rank 0] step=860, skipped=3, lr=[9.208888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:00:45,893] [INFO] [timer.py:197:stop] 0/1720, RunningAvgSamplesPerSec=5.863010438365429, CurrSamplesPerSec=5.293719976799057, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 850/5000 [2:38:21<12:43:46, 11.04s/it]
                                                       
{'loss': 0.0025, 'learning_rate': 9.208888888888889e-06, 'epoch': 20.72}

 17%|█▋        | 850/5000 [2:38:21<12:43:46, 11.04s/it][2022-12-20 13:00:56,925] [INFO] [timer.py:197:stop] 0/1722, RunningAvgSamplesPerSec=5.863027448709015, CurrSamplesPerSec=5.313869771404317, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 851/5000 [2:38:32<12:43:21, 11.04s/it][2022-12-20 13:01:07,909] [INFO] [timer.py:197:stop] 0/1724, RunningAvgSamplesPerSec=5.863062060303542, CurrSamplesPerSec=5.370356315689148, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 852/5000 [2:38:43<12:42:01, 11.02s/it][2022-12-20 13:01:19,006] [INFO] [timer.py:197:stop] 0/1726, RunningAvgSamplesPerSec=5.863026292242862, CurrSamplesPerSec=5.263951639537692, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 853/5000 [2:38:54<12:43:23, 11.04s/it][2022-12-20 13:01:30,043] [INFO] [timer.py:197:stop] 0/1728, RunningAvgSamplesPerSec=5.863027355389008, CurrSamplesPerSec=5.309631189986741, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 854/5000 [2:39:05<12:43:02, 11.04s/it][2022-12-20 13:01:41,092] [INFO] [timer.py:197:stop] 0/1730, RunningAvgSamplesPerSec=5.8630214319242855, CurrSamplesPerSec=5.297921472043439, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 855/5000 [2:39:16<12:43:00, 11.04s/it][2022-12-20 13:01:52,153] [INFO] [timer.py:197:stop] 0/1732, RunningAvgSamplesPerSec=5.863008479947144, CurrSamplesPerSec=5.317861220015355, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 856/5000 [2:39:27<12:43:09, 11.05s/it][2022-12-20 13:02:03,153] [INFO] [timer.py:197:stop] 0/1734, RunningAvgSamplesPerSec=5.863037051121418, CurrSamplesPerSec=5.308284283012356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 857/5000 [2:39:38<12:41:56, 11.03s/it][2022-12-20 13:02:14,417] [INFO] [timer.py:197:stop] 0/1736, RunningAvgSamplesPerSec=5.86289817492865, CurrSamplesPerSec=5.1192106685820615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 858/5000 [2:39:50<12:46:31, 11.10s/it][2022-12-20 13:02:25,476] [INFO] [timer.py:197:stop] 0/1738, RunningAvgSamplesPerSec=5.862887025204171, CurrSamplesPerSec=5.308131240012234, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 859/5000 [2:40:01<12:45:23, 11.09s/it][2022-12-20 13:02:36,482] [INFO] [logging.py:68:log_dist] [Rank 0] step=870, skipped=3, lr=[9.186666666666666e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:02:36,484] [INFO] [timer.py:197:stop] 0/1740, RunningAvgSamplesPerSec=5.862911774424114, CurrSamplesPerSec=5.339551537709063, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 860/5000 [2:40:12<12:43:31, 11.07s/it][2022-12-20 13:02:47,505] [INFO] [timer.py:197:stop] 0/1742, RunningAvgSamplesPerSec=5.862933619173089, CurrSamplesPerSec=5.347242294972296, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 861/5000 [2:40:23<12:42:25, 11.05s/it][2022-12-20 13:02:57,633] [INFO] [timer.py:197:stop] 0/1744, RunningAvgSamplesPerSec=5.863489812755948, CurrSamplesPerSec=5.317454179167009, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 862/5000 [2:40:38<14:05:46, 12.26s/it][2022-12-20 13:03:08,948] [INFO] [timer.py:197:stop] 0/1746, RunningAvgSamplesPerSec=5.863319598641232, CurrSamplesPerSec=5.08837024612517, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 863/5000 [2:40:49<13:46:30, 11.99s/it][2022-12-20 13:03:20,048] [INFO] [timer.py:197:stop] 0/1748, RunningAvgSamplesPerSec=5.863282290972904, CurrSamplesPerSec=5.291993628524226, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 864/5000 [2:41:00<13:27:01, 11.71s/it][2022-12-20 13:03:31,087] [INFO] [timer.py:197:stop] 0/1750, RunningAvgSamplesPerSec=5.863282070841998, CurrSamplesPerSec=5.305730313409047, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 865/5000 [2:41:11<13:12:27, 11.50s/it][2022-12-20 13:03:42,207] [INFO] [timer.py:197:stop] 0/1752, RunningAvgSamplesPerSec=5.863232725648309, CurrSamplesPerSec=5.212808407871761, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 866/5000 [2:41:22<13:05:29, 11.40s/it][2022-12-20 13:03:53,248] [INFO] [timer.py:197:stop] 0/1754, RunningAvgSamplesPerSec=5.863231257236576, CurrSamplesPerSec=5.325097212838518, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 867/5000 [2:41:33<12:58:17, 11.30s/it][2022-12-20 13:04:04,336] [INFO] [timer.py:197:stop] 0/1756, RunningAvgSamplesPerSec=5.863212342827838, CurrSamplesPerSec=5.302631334103707, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 868/5000 [2:41:44<12:52:35, 11.22s/it][2022-12-20 13:04:15,331] [INFO] [timer.py:197:stop] 0/1758, RunningAvgSamplesPerSec=5.863239374045542, CurrSamplesPerSec=5.334333183353181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 869/5000 [2:41:55<12:48:16, 11.16s/it][2022-12-20 13:04:26,365] [INFO] [logging.py:68:log_dist] [Rank 0] step=880, skipped=3, lr=[9.164444444444446e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:04:26,367] [INFO] [timer.py:197:stop] 0/1760, RunningAvgSamplesPerSec=5.863240460036503, CurrSamplesPerSec=5.318608466478786, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 870/5000 [2:42:06<12:44:43, 11.11s/it][2022-12-20 13:04:37,416] [INFO] [timer.py:197:stop] 0/1762, RunningAvgSamplesPerSec=5.863234304607188, CurrSamplesPerSec=5.2717624626641, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 871/5000 [2:42:17<12:44:22, 11.11s/it][2022-12-20 13:04:48,464] [INFO] [timer.py:197:stop] 0/1764, RunningAvgSamplesPerSec=5.863228628911783, CurrSamplesPerSec=5.319200765261345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 872/5000 [2:42:29<12:42:59, 11.09s/it][2022-12-20 13:04:59,515] [INFO] [timer.py:197:stop] 0/1766, RunningAvgSamplesPerSec=5.8632206700626375, CurrSamplesPerSec=5.317340210469133, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 873/5000 [2:42:40<12:42:34, 11.09s/it][2022-12-20 13:05:10,605] [INFO] [timer.py:197:stop] 0/1768, RunningAvgSamplesPerSec=5.863200026895385, CurrSamplesPerSec=5.322447260007576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 17%|█▋        | 874/5000 [2:42:51<12:42:02, 11.08s/it][2022-12-20 13:05:21,682] [INFO] [timer.py:197:stop] 0/1770, RunningAvgSamplesPerSec=5.863177485571984, CurrSamplesPerSec=5.3019851217980065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 875/5000 [2:43:02<12:41:24, 11.08s/it]
                                                       
{'loss': 0.0025, 'learning_rate': 9.153333333333334e-06, 'epoch': 21.34}

 18%|█▊        | 875/5000 [2:43:02<12:41:24, 11.08s/it][2022-12-20 13:05:32,774] [INFO] [timer.py:197:stop] 0/1772, RunningAvgSamplesPerSec=5.863146238342133, CurrSamplesPerSec=5.275257096494465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 876/5000 [2:43:13<12:41:52, 11.08s/it][2022-12-20 13:05:43,834] [INFO] [timer.py:197:stop] 0/1774, RunningAvgSamplesPerSec=5.863143359265502, CurrSamplesPerSec=5.3147495309620485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 877/5000 [2:43:24<12:40:47, 11.07s/it][2022-12-20 13:05:54,880] [INFO] [timer.py:197:stop] 0/1776, RunningAvgSamplesPerSec=5.863139746175419, CurrSamplesPerSec=5.309892292657005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 878/5000 [2:43:35<12:40:57, 11.08s/it][2022-12-20 13:06:06,130] [INFO] [timer.py:197:stop] 0/1778, RunningAvgSamplesPerSec=5.863113886367463, CurrSamplesPerSec=5.3098450275992395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 879/5000 [2:43:46<12:43:41, 11.12s/it][2022-12-20 13:06:17,174] [INFO] [logging.py:68:log_dist] [Rank 0] step=890, skipped=3, lr=[9.142222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:06:17,176] [INFO] [timer.py:197:stop] 0/1780, RunningAvgSamplesPerSec=5.863109362709774, CurrSamplesPerSec=5.319346014532816, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 880/5000 [2:43:57<12:41:14, 11.09s/it][2022-12-20 13:06:28,239] [INFO] [timer.py:197:stop] 0/1782, RunningAvgSamplesPerSec=5.863095666864154, CurrSamplesPerSec=5.273250006296026, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 881/5000 [2:44:08<12:44:39, 11.14s/it][2022-12-20 13:06:39,459] [INFO] [timer.py:197:stop] 0/1784, RunningAvgSamplesPerSec=5.863099138718322, CurrSamplesPerSec=5.3065108693194025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 882/5000 [2:44:20<12:42:50, 11.11s/it][2022-12-20 13:06:50,500] [INFO] [timer.py:197:stop] 0/1786, RunningAvgSamplesPerSec=5.863108168546899, CurrSamplesPerSec=5.338831523919321, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 883/5000 [2:44:31<12:41:06, 11.09s/it][2022-12-20 13:07:01,643] [INFO] [timer.py:197:stop] 0/1788, RunningAvgSamplesPerSec=5.8630457489133105, CurrSamplesPerSec=5.233432427881948, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 884/5000 [2:44:42<12:41:38, 11.10s/it][2022-12-20 13:07:12,651] [INFO] [timer.py:197:stop] 0/1790, RunningAvgSamplesPerSec=5.863064704392714, CurrSamplesPerSec=5.338598782236333, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 885/5000 [2:44:53<12:38:57, 11.07s/it][2022-12-20 13:07:23,620] [INFO] [timer.py:197:stop] 0/1792, RunningAvgSamplesPerSec=5.863116491947363, CurrSamplesPerSec=5.348120780319252, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 886/5000 [2:45:04<12:37:56, 11.05s/it][2022-12-20 13:07:34,699] [INFO] [timer.py:197:stop] 0/1794, RunningAvgSamplesPerSec=5.863102725323443, CurrSamplesPerSec=5.3160986684931775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 887/5000 [2:45:15<12:37:15, 11.05s/it][2022-12-20 13:07:45,677] [INFO] [timer.py:197:stop] 0/1796, RunningAvgSamplesPerSec=5.863139327845926, CurrSamplesPerSec=5.348327712752244, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 888/5000 [2:45:26<12:36:00, 11.03s/it][2022-12-20 13:07:56,768] [INFO] [timer.py:197:stop] 0/1798, RunningAvgSamplesPerSec=5.863108366370705, CurrSamplesPerSec=5.263477467684085, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 889/5000 [2:45:37<12:37:36, 11.06s/it][2022-12-20 13:08:07,828] [INFO] [logging.py:68:log_dist] [Rank 0] step=900, skipped=3, lr=[9.12e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:08:07,830] [INFO] [timer.py:197:stop] 0/1800, RunningAvgSamplesPerSec=5.863105117821112, CurrSamplesPerSec=5.312150441760558, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 890/5000 [2:45:48<12:36:54, 11.05s/it][2022-12-20 13:08:18,883] [INFO] [timer.py:197:stop] 0/1802, RunningAvgSamplesPerSec=5.863096549895932, CurrSamplesPerSec=5.293174670543398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 891/5000 [2:45:59<12:37:57, 11.07s/it][2022-12-20 13:08:29,997] [INFO] [timer.py:197:stop] 0/1804, RunningAvgSamplesPerSec=5.863057436659004, CurrSamplesPerSec=5.297210759384711, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 892/5000 [2:46:10<12:37:24, 11.06s/it][2022-12-20 13:08:40,971] [INFO] [timer.py:197:stop] 0/1806, RunningAvgSamplesPerSec=5.863101854772101, CurrSamplesPerSec=5.356824683031214, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 893/5000 [2:46:21<12:35:12, 11.03s/it][2022-12-20 13:08:52,356] [INFO] [timer.py:197:stop] 0/1808, RunningAvgSamplesPerSec=5.862901337791312, CurrSamplesPerSec=5.011563083772783, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 894/5000 [2:46:32<12:44:05, 11.17s/it][2022-12-20 13:09:03,465] [INFO] [timer.py:197:stop] 0/1810, RunningAvgSamplesPerSec=5.862883687042109, CurrSamplesPerSec=5.311192516655532, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 895/5000 [2:46:43<12:41:02, 11.12s/it][2022-12-20 13:09:14,500] [INFO] [timer.py:197:stop] 0/1812, RunningAvgSamplesPerSec=5.8628863066910135, CurrSamplesPerSec=5.2996468777061105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 896/5000 [2:46:55<12:38:50, 11.09s/it][2022-12-20 13:09:25,599] [INFO] [timer.py:197:stop] 0/1814, RunningAvgSamplesPerSec=5.862850856544724, CurrSamplesPerSec=5.236186476467094, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 897/5000 [2:47:06<12:39:27, 11.11s/it][2022-12-20 13:09:36,642] [INFO] [timer.py:197:stop] 0/1816, RunningAvgSamplesPerSec=5.862849452907005, CurrSamplesPerSec=5.31515721025636, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 898/5000 [2:47:17<12:37:41, 11.08s/it][2022-12-20 13:09:47,647] [INFO] [timer.py:197:stop] 0/1818, RunningAvgSamplesPerSec=5.862879615005573, CurrSamplesPerSec=5.335823373385153, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 899/5000 [2:47:28<12:39:11, 11.11s/it][2022-12-20 13:09:58,791] [INFO] [logging.py:68:log_dist] [Rank 0] step=910, skipped=3, lr=[9.097777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:09:58,793] [INFO] [timer.py:197:stop] 0/1820, RunningAvgSamplesPerSec=5.8628168502735765, CurrSamplesPerSec=5.352640279774506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 900/5000 [2:47:39<12:36:50, 11.08s/it]
                                                       
{'loss': 0.0022, 'learning_rate': 9.097777777777778e-06, 'epoch': 21.94}

 18%|█▊        | 900/5000 [2:47:39<12:36:50, 11.08s/it][2022-12-20 13:10:09,804] [INFO] [timer.py:197:stop] 0/1822, RunningAvgSamplesPerSec=5.862845374485249, CurrSamplesPerSec=5.360856965585368, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 901/5000 [2:47:50<12:34:40, 11.05s/it][2022-12-20 13:10:20,872] [INFO] [timer.py:197:stop] 0/1824, RunningAvgSamplesPerSec=5.862829010451531, CurrSamplesPerSec=5.266756511472479, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 902/5000 [2:48:01<12:34:51, 11.05s/it][2022-12-20 13:10:30,940] [INFO] [timer.py:197:stop] 0/1826, RunningAvgSamplesPerSec=5.863395446450695, CurrSamplesPerSec=6.286922146800571, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 13:10:41,924] [INFO] [timer.py:197:stop] 0/1828, RunningAvgSamplesPerSec=5.863425914746039, CurrSamplesPerSec=5.306815937987885, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 903/5000 [2:48:17<14:18:38, 12.57s/it][2022-12-20 13:10:53,007] [INFO] [timer.py:197:stop] 0/1830, RunningAvgSamplesPerSec=5.8633999137959885, CurrSamplesPerSec=5.242540463590288, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 904/5000 [2:48:28<13:47:53, 12.13s/it][2022-12-20 13:11:04,009] [INFO] [timer.py:197:stop] 0/1832, RunningAvgSamplesPerSec=5.863422339097413, CurrSamplesPerSec=5.3108505893395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 905/5000 [2:48:39<13:24:38, 11.79s/it][2022-12-20 13:11:15,205] [INFO] [timer.py:197:stop] 0/1834, RunningAvgSamplesPerSec=5.863337497886234, CurrSamplesPerSec=5.161115431980984, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 906/5000 [2:48:50<13:12:35, 11.62s/it][2022-12-20 13:11:26,525] [INFO] [timer.py:197:stop] 0/1836, RunningAvgSamplesPerSec=5.863307066460995, CurrSamplesPerSec=5.28140558438412, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 907/5000 [2:49:02<13:06:03, 11.52s/it][2022-12-20 13:11:37,565] [INFO] [timer.py:197:stop] 0/1838, RunningAvgSamplesPerSec=5.863306919315957, CurrSamplesPerSec=5.319504764826309, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 908/5000 [2:49:13<12:55:58, 11.38s/it][2022-12-20 13:11:48,636] [INFO] [logging.py:68:log_dist] [Rank 0] step=920, skipped=3, lr=[9.075555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:11:48,637] [INFO] [timer.py:197:stop] 0/1840, RunningAvgSamplesPerSec=5.863287703870412, CurrSamplesPerSec=5.281265932452559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 909/5000 [2:49:24<12:49:31, 11.29s/it][2022-12-20 13:11:59,626] [INFO] [timer.py:197:stop] 0/1842, RunningAvgSamplesPerSec=5.863316990575923, CurrSamplesPerSec=5.353972631569961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 910/5000 [2:49:35<12:43:15, 11.20s/it][2022-12-20 13:12:10,588] [INFO] [timer.py:197:stop] 0/1844, RunningAvgSamplesPerSec=5.863361869564128, CurrSamplesPerSec=5.349802490016084, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 911/5000 [2:49:46<12:38:16, 11.13s/it][2022-12-20 13:12:21,720] [INFO] [timer.py:197:stop] 0/1846, RunningAvgSamplesPerSec=5.863307776583778, CurrSamplesPerSec=5.298496203528861, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 912/5000 [2:49:57<12:38:12, 11.13s/it][2022-12-20 13:12:32,692] [INFO] [timer.py:197:stop] 0/1848, RunningAvgSamplesPerSec=5.863346925677377, CurrSamplesPerSec=5.334047837068986, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 913/5000 [2:50:08<12:34:49, 11.08s/it][2022-12-20 13:12:43,961] [INFO] [timer.py:197:stop] 0/1850, RunningAvgSamplesPerSec=5.863213557445657, CurrSamplesPerSec=5.127309835452285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 914/5000 [2:50:19<12:38:28, 11.14s/it][2022-12-20 13:12:54,998] [INFO] [timer.py:197:stop] 0/1852, RunningAvgSamplesPerSec=5.8632152600117555, CurrSamplesPerSec=5.297232084262069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 915/5000 [2:50:30<12:36:13, 11.11s/it][2022-12-20 13:13:06,064] [INFO] [timer.py:197:stop] 0/1854, RunningAvgSamplesPerSec=5.86319984146624, CurrSamplesPerSec=5.289854319628167, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 916/5000 [2:50:41<12:35:11, 11.09s/it][2022-12-20 13:13:17,034] [INFO] [timer.py:197:stop] 0/1856, RunningAvgSamplesPerSec=5.863239586770711, CurrSamplesPerSec=5.354203298256232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 917/5000 [2:50:52<12:32:28, 11.06s/it][2022-12-20 13:13:28,057] [INFO] [timer.py:197:stop] 0/1858, RunningAvgSamplesPerSec=5.863258977058892, CurrSamplesPerSec=5.342571337690337, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 918/5000 [2:51:03<12:31:34, 11.05s/it][2022-12-20 13:13:39,364] [INFO] [logging.py:68:log_dist] [Rank 0] step=930, skipped=3, lr=[9.053333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:13:39,366] [INFO] [timer.py:197:stop] 0/1860, RunningAvgSamplesPerSec=5.8631026027444095, CurrSamplesPerSec=5.064435982947557, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 919/5000 [2:51:14<12:36:44, 11.13s/it][2022-12-20 13:13:50,393] [INFO] [timer.py:197:stop] 0/1862, RunningAvgSamplesPerSec=5.863109733907907, CurrSamplesPerSec=5.305720455665611, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 920/5000 [2:51:25<12:34:31, 11.10s/it][2022-12-20 13:14:01,476] [INFO] [timer.py:197:stop] 0/1864, RunningAvgSamplesPerSec=5.863087682520712, CurrSamplesPerSec=5.297449733269956, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 921/5000 [2:51:37<12:34:05, 11.09s/it][2022-12-20 13:14:12,544] [INFO] [timer.py:197:stop] 0/1866, RunningAvgSamplesPerSec=5.863071758398854, CurrSamplesPerSec=5.330912523733726, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 922/5000 [2:51:48<12:33:24, 11.09s/it][2022-12-20 13:14:23,603] [INFO] [timer.py:197:stop] 0/1868, RunningAvgSamplesPerSec=5.863061291016458, CurrSamplesPerSec=5.2960396182983125, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 923/5000 [2:51:59<12:32:41, 11.08s/it][2022-12-20 13:14:34,743] [INFO] [timer.py:197:stop] 0/1870, RunningAvgSamplesPerSec=5.863003846766598, CurrSamplesPerSec=5.251928087358001, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 924/5000 [2:52:10<12:33:46, 11.10s/it][2022-12-20 13:14:45,814] [INFO] [timer.py:197:stop] 0/1872, RunningAvgSamplesPerSec=5.862995083904584, CurrSamplesPerSec=5.311952816459148, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 18%|█▊        | 925/5000 [2:52:21<12:33:06, 11.09s/it]
                                                       
{'loss': 0.002, 'learning_rate': 9.040000000000002e-06, 'epoch': 22.55}

 18%|█▊        | 925/5000 [2:52:21<12:33:06, 11.09s/it][2022-12-20 13:14:56,799] [INFO] [timer.py:197:stop] 0/1874, RunningAvgSamplesPerSec=5.863027955629476, CurrSamplesPerSec=5.332433428574754, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 926/5000 [2:52:32<12:30:47, 11.06s/it][2022-12-20 13:15:07,913] [INFO] [timer.py:197:stop] 0/1876, RunningAvgSamplesPerSec=5.862985534112882, CurrSamplesPerSec=5.288681635677983, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 927/5000 [2:52:43<12:31:46, 11.07s/it][2022-12-20 13:15:18,950] [INFO] [timer.py:197:stop] 0/1878, RunningAvgSamplesPerSec=5.862987719111392, CurrSamplesPerSec=5.31312196287513, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 928/5000 [2:52:54<12:30:48, 11.06s/it][2022-12-20 13:15:29,979] [INFO] [logging.py:68:log_dist] [Rank 0] step=940, skipped=3, lr=[9.031111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:15:29,991] [INFO] [timer.py:197:stop] 0/1880, RunningAvgSamplesPerSec=5.862988762777321, CurrSamplesPerSec=5.2859212330842835, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 929/5000 [2:53:05<12:30:27, 11.06s/it][2022-12-20 13:15:41,205] [INFO] [timer.py:197:stop] 0/1882, RunningAvgSamplesPerSec=5.862978868027188, CurrSamplesPerSec=5.317657481118272, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 930/5000 [2:53:16<12:33:07, 11.10s/it][2022-12-20 13:15:52,224] [INFO] [timer.py:197:stop] 0/1884, RunningAvgSamplesPerSec=5.862990450171248, CurrSamplesPerSec=5.311401225339082, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 931/5000 [2:53:27<12:31:14, 11.08s/it][2022-12-20 13:16:03,209] [INFO] [timer.py:197:stop] 0/1886, RunningAvgSamplesPerSec=5.86302138562864, CurrSamplesPerSec=5.317339789152142, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 932/5000 [2:53:38<12:29:10, 11.05s/it][2022-12-20 13:16:14,178] [INFO] [timer.py:197:stop] 0/1888, RunningAvgSamplesPerSec=5.863061789978957, CurrSamplesPerSec=5.308819268073015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 933/5000 [2:53:49<12:27:21, 11.03s/it][2022-12-20 13:16:25,199] [INFO] [timer.py:197:stop] 0/1890, RunningAvgSamplesPerSec=5.863072505625755, CurrSamplesPerSec=5.336709144626988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 934/5000 [2:54:00<12:27:04, 11.02s/it][2022-12-20 13:16:36,154] [INFO] [timer.py:197:stop] 0/1892, RunningAvgSamplesPerSec=5.86312053441808, CurrSamplesPerSec=5.343313207256025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 935/5000 [2:54:11<12:25:28, 11.00s/it][2022-12-20 13:16:47,163] [INFO] [timer.py:197:stop] 0/1894, RunningAvgSamplesPerSec=5.86313792383546, CurrSamplesPerSec=5.318227020022382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 936/5000 [2:54:22<12:25:24, 11.01s/it][2022-12-20 13:16:58,167] [INFO] [timer.py:197:stop] 0/1896, RunningAvgSamplesPerSec=5.863157616650631, CurrSamplesPerSec=5.317288178325738, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▊        | 937/5000 [2:54:33<12:25:12, 11.00s/it][2022-12-20 13:17:09,171] [INFO] [timer.py:197:stop] 0/1898, RunningAvgSamplesPerSec=5.863177029485665, CurrSamplesPerSec=5.32252830957256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 938/5000 [2:54:44<12:25:00, 11.00s/it][2022-12-20 13:17:20,230] [INFO] [logging.py:68:log_dist] [Rank 0] step=950, skipped=3, lr=[9.008888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:17:20,232] [INFO] [timer.py:197:stop] 0/1900, RunningAvgSamplesPerSec=5.863170584291241, CurrSamplesPerSec=5.314059123412461, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 939/5000 [2:54:55<12:25:57, 11.02s/it][2022-12-20 13:17:31,281] [INFO] [timer.py:197:stop] 0/1902, RunningAvgSamplesPerSec=5.863164082986453, CurrSamplesPerSec=5.312269234256793, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 940/5000 [2:55:06<12:26:20, 11.03s/it][2022-12-20 13:17:42,501] [INFO] [timer.py:197:stop] 0/1904, RunningAvgSamplesPerSec=5.863061769409012, CurrSamplesPerSec=5.1656274363425325, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 941/5000 [2:55:18<12:30:01, 11.09s/it][2022-12-20 13:17:53,541] [INFO] [timer.py:197:stop] 0/1906, RunningAvgSamplesPerSec=5.863061823980693, CurrSamplesPerSec=5.332735340629766, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 942/5000 [2:55:29<12:28:52, 11.07s/it][2022-12-20 13:18:04,566] [INFO] [timer.py:197:stop] 0/1908, RunningAvgSamplesPerSec=5.863071238025075, CurrSamplesPerSec=5.329390581693254, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 943/5000 [2:55:40<12:27:43, 11.06s/it][2022-12-20 13:18:14,811] [INFO] [timer.py:197:stop] 0/1910, RunningAvgSamplesPerSec=5.8635127137872285, CurrSamplesPerSec=5.189843449059289, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 944/5000 [2:55:55<13:51:00, 12.29s/it][2022-12-20 13:18:25,828] [INFO] [timer.py:197:stop] 0/1912, RunningAvgSamplesPerSec=5.86352408107808, CurrSamplesPerSec=5.310804357924676, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 945/5000 [2:56:06<13:25:11, 11.91s/it][2022-12-20 13:18:36,875] [INFO] [timer.py:197:stop] 0/1914, RunningAvgSamplesPerSec=5.863528622901098, CurrSamplesPerSec=5.29624212199257, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 946/5000 [2:56:17<13:07:12, 11.65s/it][2022-12-20 13:18:47,876] [INFO] [timer.py:197:stop] 0/1916, RunningAvgSamplesPerSec=5.863558921906664, CurrSamplesPerSec=5.342188785369017, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 947/5000 [2:56:28<12:54:26, 11.46s/it][2022-12-20 13:18:58,920] [INFO] [timer.py:197:stop] 0/1918, RunningAvgSamplesPerSec=5.863555755300227, CurrSamplesPerSec=5.315760109333357, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 948/5000 [2:56:39<12:45:37, 11.34s/it][2022-12-20 13:19:09,946] [INFO] [logging.py:68:log_dist] [Rank 0] step=960, skipped=3, lr=[8.986666666666666e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:19:09,947] [INFO] [timer.py:197:stop] 0/1920, RunningAvgSamplesPerSec=5.863561495748736, CurrSamplesPerSec=5.325827051810397, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 949/5000 [2:56:50<12:39:05, 11.24s/it][2022-12-20 13:19:20,925] [INFO] [timer.py:197:stop] 0/1922, RunningAvgSamplesPerSec=5.8635967583565085, CurrSamplesPerSec=5.368065381599439, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 950/5000 [2:57:01<12:33:04, 11.16s/it]
                                                       
{'loss': 0.002, 'learning_rate': 8.984444444444446e-06, 'epoch': 23.17}

 19%|█▉        | 950/5000 [2:57:01<12:33:04, 11.16s/it][2022-12-20 13:19:31,926] [INFO] [timer.py:197:stop] 0/1924, RunningAvgSamplesPerSec=5.863618389029232, CurrSamplesPerSec=5.327956433711569, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 951/5000 [2:57:12<12:29:41, 11.11s/it][2022-12-20 13:19:42,973] [INFO] [timer.py:197:stop] 0/1926, RunningAvgSamplesPerSec=5.863613155272715, CurrSamplesPerSec=5.283352128350707, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 952/5000 [2:57:23<12:28:15, 11.09s/it][2022-12-20 13:19:53,977] [INFO] [timer.py:197:stop] 0/1928, RunningAvgSamplesPerSec=5.863632472210253, CurrSamplesPerSec=5.323241609971915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 953/5000 [2:57:34<12:26:15, 11.06s/it][2022-12-20 13:20:04,956] [INFO] [timer.py:197:stop] 0/1930, RunningAvgSamplesPerSec=5.863674730085934, CurrSamplesPerSec=5.342153488693842, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 954/5000 [2:57:45<12:24:52, 11.05s/it][2022-12-20 13:20:15,940] [INFO] [timer.py:197:stop] 0/1932, RunningAvgSamplesPerSec=5.863705267630054, CurrSamplesPerSec=5.360862960970324, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 955/5000 [2:57:56<12:23:46, 11.03s/it][2022-12-20 13:20:27,005] [INFO] [timer.py:197:stop] 0/1934, RunningAvgSamplesPerSec=5.863690659169831, CurrSamplesPerSec=5.303668535574321, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 956/5000 [2:58:07<12:22:34, 11.02s/it][2022-12-20 13:20:38,116] [INFO] [timer.py:197:stop] 0/1936, RunningAvgSamplesPerSec=5.863650796465589, CurrSamplesPerSec=5.192637982124209, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 957/5000 [2:58:18<12:24:17, 11.05s/it][2022-12-20 13:20:49,053] [INFO] [timer.py:197:stop] 0/1938, RunningAvgSamplesPerSec=5.863706782263476, CurrSamplesPerSec=5.343985917311587, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 958/5000 [2:58:29<12:22:58, 11.03s/it][2022-12-20 13:21:00,080] [INFO] [logging.py:68:log_dist] [Rank 0] step=970, skipped=3, lr=[8.964444444444446e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:21:00,082] [INFO] [timer.py:197:stop] 0/1940, RunningAvgSamplesPerSec=5.863711545723053, CurrSamplesPerSec=5.308376239017819, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 959/5000 [2:58:40<12:24:34, 11.06s/it][2022-12-20 13:21:11,499] [INFO] [timer.py:197:stop] 0/1942, RunningAvgSamplesPerSec=5.863669485556379, CurrSamplesPerSec=5.306082070056966, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 960/5000 [2:58:52<12:30:07, 11.14s/it][2022-12-20 13:21:22,542] [INFO] [timer.py:197:stop] 0/1944, RunningAvgSamplesPerSec=5.863666266486689, CurrSamplesPerSec=5.304695659968548, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 961/5000 [2:59:03<12:27:14, 11.10s/it][2022-12-20 13:21:33,586] [INFO] [timer.py:197:stop] 0/1946, RunningAvgSamplesPerSec=5.863664862124643, CurrSamplesPerSec=5.275694614419871, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 962/5000 [2:59:14<12:29:32, 11.14s/it][2022-12-20 13:21:44,765] [INFO] [timer.py:197:stop] 0/1948, RunningAvgSamplesPerSec=5.863657878493431, CurrSamplesPerSec=5.311903412431323, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 963/5000 [2:59:25<12:27:01, 11.10s/it][2022-12-20 13:21:55,821] [INFO] [timer.py:197:stop] 0/1950, RunningAvgSamplesPerSec=5.86364849928639, CurrSamplesPerSec=5.283084895163707, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 964/5000 [2:59:36<12:26:41, 11.10s/it][2022-12-20 13:22:06,931] [INFO] [timer.py:197:stop] 0/1952, RunningAvgSamplesPerSec=5.863608723714869, CurrSamplesPerSec=5.269590028614178, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 965/5000 [2:59:47<12:26:47, 11.10s/it][2022-12-20 13:22:18,011] [INFO] [timer.py:197:stop] 0/1954, RunningAvgSamplesPerSec=5.863595177958572, CurrSamplesPerSec=5.299245131033772, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 966/5000 [2:59:58<12:26:07, 11.10s/it][2022-12-20 13:22:29,102] [INFO] [timer.py:197:stop] 0/1956, RunningAvgSamplesPerSec=5.863566994906607, CurrSamplesPerSec=5.291377959330684, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 967/5000 [3:00:09<12:25:41, 11.09s/it][2022-12-20 13:22:40,150] [INFO] [timer.py:197:stop] 0/1958, RunningAvgSamplesPerSec=5.863561373602597, CurrSamplesPerSec=5.323026480898808, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 968/5000 [3:00:20<12:23:42, 11.07s/it][2022-12-20 13:22:51,180] [INFO] [logging.py:68:log_dist] [Rank 0] step=980, skipped=3, lr=[8.942222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:22:51,182] [INFO] [timer.py:197:stop] 0/1960, RunningAvgSamplesPerSec=5.863564922920538, CurrSamplesPerSec=5.299039050259681, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 969/5000 [3:00:31<12:23:12, 11.06s/it][2022-12-20 13:23:02,242] [INFO] [timer.py:197:stop] 0/1962, RunningAvgSamplesPerSec=5.863552959941312, CurrSamplesPerSec=5.292105469953978, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 970/5000 [3:00:42<12:23:49, 11.07s/it][2022-12-20 13:23:13,328] [INFO] [timer.py:197:stop] 0/1964, RunningAvgSamplesPerSec=5.863536339381892, CurrSamplesPerSec=5.30666088150973, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 971/5000 [3:00:53<12:23:27, 11.07s/it][2022-12-20 13:23:24,506] [INFO] [timer.py:197:stop] 0/1966, RunningAvgSamplesPerSec=5.863460257032616, CurrSamplesPerSec=5.208675280517253, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 972/5000 [3:01:05<12:25:50, 11.11s/it][2022-12-20 13:23:35,552] [INFO] [timer.py:197:stop] 0/1968, RunningAvgSamplesPerSec=5.8634571033583525, CurrSamplesPerSec=5.342969046042548, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 973/5000 [3:01:16<12:22:34, 11.06s/it][2022-12-20 13:23:46,543] [INFO] [timer.py:197:stop] 0/1970, RunningAvgSamplesPerSec=5.86348337294498, CurrSamplesPerSec=5.312444594306136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 19%|█▉        | 974/5000 [3:01:27<12:21:53, 11.06s/it][2022-12-20 13:23:57,619] [INFO] [timer.py:197:stop] 0/1972, RunningAvgSamplesPerSec=5.863462849558467, CurrSamplesPerSec=5.279448221845682, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 975/5000 [3:01:38<12:22:37, 11.07s/it]
                                                       
{'loss': 0.0021, 'learning_rate': 8.92888888888889e-06, 'epoch': 23.77}

 20%|█▉        | 975/5000 [3:01:38<12:22:37, 11.07s/it][2022-12-20 13:24:08,678] [INFO] [timer.py:197:stop] 0/1974, RunningAvgSamplesPerSec=5.863462244106491, CurrSamplesPerSec=5.319093888787919, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 976/5000 [3:01:49<12:21:44, 11.06s/it][2022-12-20 13:24:19,687] [INFO] [timer.py:197:stop] 0/1976, RunningAvgSamplesPerSec=5.863479226724475, CurrSamplesPerSec=5.341819894361356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 977/5000 [3:02:00<12:21:23, 11.06s/it][2022-12-20 13:24:30,772] [INFO] [timer.py:197:stop] 0/1978, RunningAvgSamplesPerSec=5.863454485559493, CurrSamplesPerSec=5.312210573227383, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 978/5000 [3:02:11<12:20:01, 11.04s/it][2022-12-20 13:24:41,799] [INFO] [logging.py:68:log_dist] [Rank 0] step=990, skipped=3, lr=[8.920000000000001e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:24:41,801] [INFO] [timer.py:197:stop] 0/1980, RunningAvgSamplesPerSec=5.86345958272462, CurrSamplesPerSec=5.283817407359102, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 979/5000 [3:02:22<12:20:20, 11.05s/it][2022-12-20 13:24:52,994] [INFO] [timer.py:197:stop] 0/1982, RunningAvgSamplesPerSec=5.863377421965584, CurrSamplesPerSec=5.176719901928386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 980/5000 [3:02:33<12:27:37, 11.16s/it][2022-12-20 13:25:04,236] [INFO] [timer.py:197:stop] 0/1984, RunningAvgSamplesPerSec=5.863396592956017, CurrSamplesPerSec=5.326128850293444, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 981/5000 [3:02:44<12:24:58, 11.12s/it][2022-12-20 13:25:15,247] [INFO] [timer.py:197:stop] 0/1986, RunningAvgSamplesPerSec=5.863411542097881, CurrSamplesPerSec=5.348597963768394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 982/5000 [3:02:55<12:21:47, 11.08s/it][2022-12-20 13:25:26,322] [INFO] [timer.py:197:stop] 0/1988, RunningAvgSamplesPerSec=5.863391495882291, CurrSamplesPerSec=5.258195109451015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 983/5000 [3:03:06<12:21:11, 11.07s/it][2022-12-20 13:25:37,297] [INFO] [timer.py:197:stop] 0/1990, RunningAvgSamplesPerSec=5.86342572742155, CurrSamplesPerSec=5.329679451005927, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 984/5000 [3:03:17<12:19:44, 11.05s/it][2022-12-20 13:25:47,395] [INFO] [timer.py:197:stop] 0/1992, RunningAvgSamplesPerSec=5.863928435581411, CurrSamplesPerSec=6.270034036836022, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 13:25:58,727] [INFO] [timer.py:197:stop] 0/1994, RunningAvgSamplesPerSec=5.863767935245963, CurrSamplesPerSec=5.3137359707889225, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 985/5000 [3:03:34<14:08:43, 12.68s/it][2022-12-20 13:26:09,797] [INFO] [timer.py:197:stop] 0/1996, RunningAvgSamplesPerSec=5.863760720224446, CurrSamplesPerSec=5.309494662210023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 986/5000 [3:03:45<13:36:07, 12.20s/it][2022-12-20 13:26:21,135] [INFO] [timer.py:197:stop] 0/1998, RunningAvgSamplesPerSec=5.863599749893698, CurrSamplesPerSec=5.064788771299256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 987/5000 [3:03:56<13:18:38, 11.94s/it][2022-12-20 13:26:32,211] [INFO] [logging.py:68:log_dist] [Rank 0] step=1000, skipped=3, lr=[8.897777777777779e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:26:32,212] [INFO] [timer.py:197:stop] 0/2000, RunningAvgSamplesPerSec=5.863578399609565, CurrSamplesPerSec=5.283350048610635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 988/5000 [3:04:07<13:01:07, 11.68s/it][2022-12-20 13:26:43,198] [INFO] [timer.py:197:stop] 0/2002, RunningAvgSamplesPerSec=5.863609485060845, CurrSamplesPerSec=5.317708888426303, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 989/5000 [3:04:18<12:46:58, 11.47s/it][2022-12-20 13:26:54,330] [INFO] [timer.py:197:stop] 0/2004, RunningAvgSamplesPerSec=5.863559714576657, CurrSamplesPerSec=5.31504270856817, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 990/5000 [3:04:29<12:39:56, 11.37s/it][2022-12-20 13:27:05,293] [INFO] [timer.py:197:stop] 0/2006, RunningAvgSamplesPerSec=5.863607463739625, CurrSamplesPerSec=5.3863316891139075, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 991/5000 [3:04:40<12:31:34, 11.25s/it][2022-12-20 13:27:16,359] [INFO] [timer.py:197:stop] 0/2008, RunningAvgSamplesPerSec=5.863591932184642, CurrSamplesPerSec=5.220822026936195, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 992/5000 [3:04:51<12:27:44, 11.19s/it][2022-12-20 13:27:27,349] [INFO] [timer.py:197:stop] 0/2010, RunningAvgSamplesPerSec=5.863618031665682, CurrSamplesPerSec=5.328709905116629, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 993/5000 [3:05:02<12:23:29, 11.13s/it][2022-12-20 13:27:38,391] [INFO] [timer.py:197:stop] 0/2012, RunningAvgSamplesPerSec=5.863625580070775, CurrSamplesPerSec=5.336957637708522, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 994/5000 [3:05:13<12:21:27, 11.11s/it][2022-12-20 13:27:49,465] [INFO] [timer.py:197:stop] 0/2014, RunningAvgSamplesPerSec=5.86360649181715, CurrSamplesPerSec=5.310163715411395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 995/5000 [3:05:25<12:20:39, 11.10s/it][2022-12-20 13:28:00,503] [INFO] [timer.py:197:stop] 0/2016, RunningAvgSamplesPerSec=5.863606507511726, CurrSamplesPerSec=5.31390048759089, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 996/5000 [3:05:36<12:19:19, 11.08s/it][2022-12-20 13:28:11,543] [INFO] [timer.py:197:stop] 0/2018, RunningAvgSamplesPerSec=5.863605652138122, CurrSamplesPerSec=5.286935038374454, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 997/5000 [3:05:47<12:18:21, 11.07s/it][2022-12-20 13:28:22,513] [INFO] [logging.py:68:log_dist] [Rank 0] step=1010, skipped=3, lr=[8.875555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:28:22,515] [INFO] [timer.py:197:stop] 0/2020, RunningAvgSamplesPerSec=5.863641384687522, CurrSamplesPerSec=5.349640860118194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 998/5000 [3:05:58<12:16:15, 11.04s/it][2022-12-20 13:28:33,504] [INFO] [timer.py:197:stop] 0/2022, RunningAvgSamplesPerSec=5.863667560480435, CurrSamplesPerSec=5.35376056390389, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|█▉        | 999/5000 [3:06:09<12:15:05, 11.02s/it][2022-12-20 13:28:44,577] [INFO] [timer.py:197:stop] 0/2024, RunningAvgSamplesPerSec=5.863656270334864, CurrSamplesPerSec=5.318975844814846, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1000/5000 [3:06:20<12:15:53, 11.04s/it]
                                                        
{'loss': 0.0022, 'learning_rate': 8.871111111111111e-06, 'epoch': 24.39}

 20%|██        | 1000/5000 [3:06:20<12:15:53, 11.04s/it][INFO|trainer.py:2955] 2022-12-20 13:28:44,584 >> ***** Running Evaluation *****
[INFO|trainer.py:2957] 2022-12-20 13:28:44,584 >>   Num examples = 1325
[INFO|trainer.py:2960] 2022-12-20 13:28:44,584 >>   Batch size = 8


  0%|          | 0/166 [00:00<?, ?it/s]

  1%|          | 2/166 [00:05<08:02,  2.94s/it]

  2%|▏         | 3/166 [00:11<11:11,  4.12s/it]

  2%|▏         | 4/166 [00:17<13:05,  4.85s/it]

  3%|▎         | 5/166 [00:24<15:12,  5.67s/it]

  4%|▎         | 6/166 [00:32<16:34,  6.22s/it]

  4%|▍         | 7/166 [00:38<16:35,  6.26s/it]

  5%|▍         | 8/166 [00:44<16:17,  6.19s/it]

  5%|▌         | 9/166 [00:50<15:57,  6.10s/it]

  6%|▌         | 10/166 [00:56<15:32,  5.98s/it]

  7%|▋         | 11/166 [01:01<15:10,  5.88s/it]

  7%|▋         | 12/166 [01:07<15:11,  5.92s/it]

  8%|▊         | 13/166 [01:14<15:34,  6.11s/it]

  8%|▊         | 14/166 [01:21<16:05,  6.35s/it]

  9%|▉         | 15/166 [01:27<15:27,  6.14s/it]

 10%|▉         | 16/166 [01:34<16:17,  6.51s/it]

 10%|█         | 17/166 [01:41<16:20,  6.58s/it]

 11%|█         | 18/166 [01:47<15:56,  6.47s/it]

 11%|█▏        | 19/166 [01:53<15:21,  6.27s/it]

 12%|█▏        | 20/166 [02:04<18:37,  7.65s/it]

 13%|█▎        | 21/166 [02:11<18:12,  7.53s/it]

 13%|█▎        | 22/166 [02:18<17:55,  7.47s/it]

 14%|█▍        | 23/166 [02:26<17:52,  7.50s/it]

 14%|█▍        | 24/166 [02:32<17:00,  7.19s/it]

 15%|█▌        | 25/166 [02:40<17:06,  7.28s/it]

 16%|█▌        | 26/166 [02:47<16:50,  7.22s/it]

 16%|█▋        | 27/166 [02:54<16:59,  7.33s/it]

 17%|█▋        | 28/166 [03:00<16:00,  6.96s/it]

 17%|█▋        | 29/166 [03:09<16:42,  7.32s/it]

 18%|█▊        | 30/166 [03:15<16:09,  7.13s/it]

 19%|█▊        | 31/166 [03:23<16:13,  7.21s/it]

 19%|█▉        | 32/166 [03:30<16:25,  7.36s/it]

 20%|█▉        | 33/166 [03:38<16:22,  7.39s/it]

 20%|██        | 34/166 [03:44<15:36,  7.10s/it]

 21%|██        | 35/166 [03:52<16:03,  7.35s/it]

 22%|██▏       | 36/166 [04:00<15:57,  7.37s/it]

 22%|██▏       | 37/166 [04:06<15:20,  7.13s/it]

 23%|██▎       | 38/166 [04:15<16:11,  7.59s/it]

 23%|██▎       | 39/166 [04:22<15:38,  7.39s/it]

 24%|██▍       | 40/166 [04:29<15:43,  7.49s/it]

 25%|██▍       | 41/166 [04:36<15:03,  7.23s/it]

 25%|██▌       | 42/166 [04:43<14:32,  7.04s/it]

 26%|██▌       | 43/166 [04:49<14:00,  6.84s/it]

 27%|██▋       | 44/166 [04:58<15:01,  7.39s/it]

 27%|██▋       | 45/166 [05:06<15:19,  7.60s/it]

 28%|██▊       | 46/166 [05:16<16:40,  8.34s/it]

 28%|██▊       | 47/166 [05:24<16:30,  8.32s/it]

 29%|██▉       | 48/166 [05:31<15:40,  7.97s/it]

 30%|██▉       | 49/166 [05:37<14:19,  7.34s/it]

 30%|███       | 50/166 [05:46<14:50,  7.68s/it]

 31%|███       | 51/166 [05:53<14:37,  7.63s/it]

 31%|███▏      | 52/166 [06:01<14:25,  7.59s/it]

 32%|███▏      | 53/166 [06:08<14:03,  7.46s/it]

 33%|███▎      | 54/166 [06:16<14:16,  7.64s/it]

 33%|███▎      | 55/166 [06:22<13:13,  7.15s/it]

 34%|███▎      | 56/166 [06:28<12:31,  6.83s/it]

 34%|███▍      | 57/166 [06:36<13:05,  7.21s/it]

 35%|███▍      | 58/166 [06:42<12:04,  6.71s/it]

 36%|███▌      | 59/166 [06:49<12:05,  6.78s/it]

 36%|███▌      | 60/166 [06:56<12:12,  6.91s/it]

 37%|███▋      | 61/166 [07:05<13:12,  7.55s/it]

 37%|███▋      | 62/166 [07:12<13:03,  7.54s/it]

 38%|███▊      | 63/166 [07:20<13:00,  7.58s/it]

 39%|███▊      | 64/166 [07:29<13:44,  8.08s/it]

 39%|███▉      | 65/166 [07:38<13:43,  8.15s/it]

 40%|███▉      | 66/166 [07:45<13:06,  7.86s/it]

 40%|████      | 67/166 [07:52<12:30,  7.58s/it]

 41%|████      | 68/166 [07:57<11:10,  6.85s/it]

 42%|████▏     | 69/166 [08:04<11:07,  6.88s/it]

 42%|████▏     | 70/166 [08:12<11:49,  7.40s/it]

 43%|████▎     | 71/166 [08:20<11:56,  7.54s/it]

 43%|████▎     | 72/166 [08:27<11:18,  7.22s/it]

 44%|████▍     | 73/166 [08:33<10:50,  6.99s/it]

 45%|████▍     | 74/166 [08:40<10:30,  6.85s/it]

 45%|████▌     | 75/166 [08:46<10:10,  6.71s/it]

 46%|████▌     | 76/166 [08:53<10:21,  6.90s/it]

 46%|████▋     | 77/166 [08:59<09:47,  6.60s/it]

 47%|████▋     | 78/166 [09:06<09:53,  6.74s/it]

 48%|████▊     | 79/166 [09:15<10:30,  7.25s/it]

 48%|████▊     | 80/166 [09:23<10:51,  7.57s/it]

 49%|████▉     | 81/166 [09:31<10:49,  7.64s/it]

 49%|████▉     | 82/166 [09:38<10:32,  7.53s/it]

 50%|█████     | 83/166 [09:44<09:31,  6.89s/it]

 51%|█████     | 84/166 [09:50<09:11,  6.73s/it]

 51%|█████     | 85/166 [09:57<09:19,  6.91s/it]

 52%|█████▏    | 86/166 [10:06<09:50,  7.38s/it]

 52%|█████▏    | 87/166 [10:13<09:48,  7.45s/it]

 53%|█████▎    | 88/166 [10:20<09:22,  7.21s/it]

 54%|█████▎    | 89/166 [10:27<09:16,  7.23s/it]

 54%|█████▍    | 90/166 [10:34<08:53,  7.02s/it]

 55%|█████▍    | 91/166 [10:41<08:39,  6.92s/it]

 55%|█████▌    | 92/166 [10:47<08:12,  6.65s/it]

 56%|█████▌    | 93/166 [10:55<08:35,  7.06s/it]

 57%|█████▋    | 94/166 [11:05<09:35,  7.99s/it]

 57%|█████▋    | 95/166 [11:10<08:24,  7.10s/it]

 58%|█████▊    | 96/166 [11:16<08:05,  6.93s/it]

 58%|█████▊    | 97/166 [11:26<09:04,  7.90s/it]

 59%|█████▉    | 98/166 [11:36<09:37,  8.49s/it]

 60%|█████▉    | 99/166 [11:44<09:07,  8.17s/it]

 60%|██████    | 100/166 [11:50<08:21,  7.60s/it]

 61%|██████    | 101/166 [11:57<07:52,  7.27s/it]

 61%|██████▏   | 102/166 [12:06<08:23,  7.87s/it]

 62%|██████▏   | 103/166 [12:15<08:35,  8.18s/it]

 63%|██████▎   | 104/166 [12:22<08:08,  7.88s/it]

 63%|██████▎   | 105/166 [12:30<08:12,  8.07s/it]

 64%|██████▍   | 106/166 [12:40<08:38,  8.65s/it]

 64%|██████▍   | 107/166 [12:50<08:44,  8.89s/it]

 65%|██████▌   | 108/166 [12:55<07:34,  7.84s/it]

 66%|██████▌   | 109/166 [13:02<07:09,  7.54s/it]

 66%|██████▋   | 110/166 [13:09<07:00,  7.51s/it]

 67%|██████▋   | 111/166 [13:15<06:25,  7.01s/it]

 67%|██████▋   | 112/166 [13:23<06:23,  7.11s/it]

 68%|██████▊   | 113/166 [13:32<06:44,  7.64s/it]

 69%|██████▊   | 114/166 [13:39<06:35,  7.61s/it]

 69%|██████▉   | 115/166 [13:46<06:10,  7.26s/it]

 70%|██████▉   | 116/166 [13:52<05:48,  6.97s/it]

 70%|███████   | 117/166 [14:00<05:54,  7.23s/it]

 71%|███████   | 118/166 [14:08<05:56,  7.43s/it]

 72%|███████▏  | 119/166 [14:15<05:53,  7.53s/it]

 72%|███████▏  | 120/166 [14:24<05:55,  7.73s/it]

 73%|███████▎  | 121/166 [14:32<06:01,  8.04s/it]

 73%|███████▎  | 122/166 [14:40<05:48,  7.92s/it]

 74%|███████▍  | 123/166 [14:47<05:34,  7.78s/it]

 75%|███████▍  | 124/166 [14:54<05:17,  7.57s/it]

 75%|███████▌  | 125/166 [15:02<05:14,  7.66s/it]

 76%|███████▌  | 126/166 [15:11<05:18,  7.96s/it]

 77%|███████▋  | 127/166 [15:16<04:36,  7.08s/it]

 77%|███████▋  | 128/166 [15:26<05:06,  8.07s/it]

 78%|███████▊  | 129/166 [15:34<04:58,  8.07s/it]

 78%|███████▊  | 130/166 [15:41<04:38,  7.73s/it]

 79%|███████▉  | 131/166 [15:48<04:23,  7.52s/it]

 80%|███████▉  | 132/166 [15:57<04:24,  7.79s/it]

 80%|████████  | 133/166 [16:05<04:17,  7.80s/it]

 81%|████████  | 134/166 [16:11<03:57,  7.42s/it]

 81%|████████▏ | 135/166 [16:17<03:37,  7.00s/it]

 82%|████████▏ | 136/166 [16:28<04:03,  8.12s/it]

 83%|████████▎ | 137/166 [16:37<04:03,  8.40s/it]

 83%|████████▎ | 138/166 [16:47<04:08,  8.89s/it]

 84%|████████▎ | 139/166 [16:56<03:57,  8.78s/it]

 84%|████████▍ | 140/166 [17:02<03:28,  8.04s/it]

 85%|████████▍ | 141/166 [17:08<03:06,  7.44s/it]

 86%|████████▌ | 142/166 [17:15<02:56,  7.35s/it]

 86%|████████▌ | 143/166 [17:21<02:39,  6.92s/it]

 87%|████████▋ | 144/166 [17:30<02:44,  7.50s/it]

 87%|████████▋ | 145/166 [17:40<02:53,  8.26s/it]

 88%|████████▊ | 146/166 [17:50<02:53,  8.67s/it]

 89%|████████▊ | 147/166 [17:58<02:43,  8.59s/it]

 89%|████████▉ | 148/166 [18:06<02:33,  8.55s/it]

 90%|████████▉ | 149/166 [18:16<02:30,  8.88s/it]

 90%|█████████ | 150/166 [18:26<02:25,  9.12s/it]

 91%|█████████ | 151/166 [18:34<02:11,  8.77s/it]

 92%|█████████▏| 152/166 [18:40<01:50,  7.90s/it]

 92%|█████████▏| 153/166 [18:46<01:38,  7.58s/it]

 93%|█████████▎| 154/166 [18:54<01:30,  7.50s/it]

 93%|█████████▎| 155/166 [19:01<01:23,  7.58s/it]

 94%|█████████▍| 156/166 [19:07<01:10,  7.07s/it]

 95%|█████████▍| 157/166 [19:14<01:03,  7.07s/it]

 95%|█████████▌| 158/166 [19:20<00:52,  6.59s/it]

 96%|█████████▌| 159/166 [19:25<00:42,  6.07s/it]

 96%|█████████▋| 160/166 [19:30<00:34,  5.79s/it]

 97%|█████████▋| 161/166 [19:36<00:29,  5.88s/it]

 98%|█████████▊| 162/166 [19:43<00:25,  6.29s/it]

 98%|█████████▊| 163/166 [19:50<00:19,  6.55s/it]

 99%|█████████▉| 164/166 [19:57<00:12,  6.44s/it]

 99%|█████████▉| 165/166 [20:02<00:06,  6.27s/it]

100%|██████████| 166/166 [20:07<00:00,  5.63s/it]
                                                        

                                                 
{'eval_loss': 0.2181396484375, 'eval_wer': 8.780685438573874, 'eval_runtime': 1286.5853, 'eval_samples_per_second': 1.03, 'eval_steps_per_second': 0.129, 'epoch': 24.39}

 20%|██        | 1000/5000 [3:27:46<12:15:53, 11.04s/it]

100%|██████████| 166/166 [21:20<00:00,  5.63s/it]

                                                 [INFO|trainer.py:2700] 2022-12-20 13:50:11,171 >> Saving model checkpoint to ./checkpoint-1000
[INFO|configuration_utils.py:447] 2022-12-20 13:50:11,172 >> Configuration saved in ./checkpoint-1000/config.json
[INFO|modeling_utils.py:1680] 2022-12-20 13:50:15,406 >> Model weights saved in ./checkpoint-1000/pytorch_model.bin
[INFO|feature_extraction_utils.py:368] 2022-12-20 13:50:15,421 >> Feature extractor saved in ./checkpoint-1000/preprocessor_config.json
[2022-12-20 13:50:15,422] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step1012 is begin to save!
/home/milan/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
[2022-12-20 13:50:15,432] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-1000/global_step1012/mp_rank_00_model_states.pt
[2022-12-20 13:50:15,432] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-1000/global_step1012/mp_rank_00_model_states.pt...
[2022-12-20 13:50:18,927] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-1000/global_step1012/mp_rank_00_model_states.pt.
[2022-12-20 13:50:18,929] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-1000/global_step1012/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2022-12-20 13:50:34,343] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-1000/global_step1012/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2022-12-20 13:50:34,343] [INFO] [engine.py:3269:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-1000/global_step1012/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2022-12-20 13:50:34,343] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1012 is ready now!
[INFO|feature_extraction_utils.py:368] 2022-12-20 13:50:36,874 >> Feature extractor saved in ./preprocessor_config.json
[2022-12-20 13:52:34,361] [INFO] [timer.py:197:stop] 0/2026, RunningAvgSamplesPerSec=5.8634860661683135, CurrSamplesPerSec=5.058270704673658, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1001/5000 [3:30:09<485:03:32, 436.66s/it][2022-12-20 13:52:45,337] [INFO] [timer.py:197:stop] 0/2028, RunningAvgSamplesPerSec=5.863520996769987, CurrSamplesPerSec=5.3502200435821905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1002/5000 [3:30:20<343:06:46, 308.96s/it][2022-12-20 13:52:56,312] [INFO] [timer.py:197:stop] 0/2030, RunningAvgSamplesPerSec=5.8635547211988195, CurrSamplesPerSec=5.317756293587114, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1003/5000 [3:30:31<243:46:28, 219.56s/it][2022-12-20 13:53:07,250] [INFO] [timer.py:197:stop] 0/2032, RunningAvgSamplesPerSec=5.863606636750225, CurrSamplesPerSec=5.362122717423734, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1004/5000 [3:30:42<174:14:31, 156.97s/it][2022-12-20 13:53:18,340] [INFO] [timer.py:197:stop] 0/2034, RunningAvgSamplesPerSec=5.86358959190176, CurrSamplesPerSec=5.306349746791664, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1005/5000 [3:30:53<125:37:51, 113.21s/it][2022-12-20 13:53:29,346] [INFO] [timer.py:197:stop] 0/2036, RunningAvgSamplesPerSec=5.863607921293516, CurrSamplesPerSec=5.311056749218157, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1006/5000 [3:31:04<91:34:58, 82.55s/it]  [2022-12-20 13:53:40,440] [INFO] [timer.py:197:stop] 0/2038, RunningAvgSamplesPerSec=5.863578509413674, CurrSamplesPerSec=5.243325682683028, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1007/5000 [3:31:16<67:47:00, 61.11s/it][2022-12-20 13:53:51,433] [INFO] [logging.py:68:log_dist] [Rank 0] step=1020, skipped=3, lr=[8.853333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:53:51,435] [INFO] [timer.py:197:stop] 0/2040, RunningAvgSamplesPerSec=5.863603336779432, CurrSamplesPerSec=5.34758721999845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1008/5000 [3:31:27<51:05:38, 46.08s/it][2022-12-20 13:54:02,491] [INFO] [timer.py:197:stop] 0/2042, RunningAvgSamplesPerSec=5.863593984023428, CurrSamplesPerSec=5.295950596806758, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1009/5000 [3:31:38<39:26:01, 35.57s/it][2022-12-20 13:54:13,532] [INFO] [timer.py:197:stop] 0/2044, RunningAvgSamplesPerSec=5.86359209027383, CurrSamplesPerSec=5.287658827953365, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1010/5000 [3:31:49<31:16:05, 28.21s/it][2022-12-20 13:54:24,560] [INFO] [timer.py:197:stop] 0/2046, RunningAvgSamplesPerSec=5.863597384976187, CurrSamplesPerSec=5.348200695684302, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1011/5000 [3:32:00<25:32:52, 23.06s/it][2022-12-20 13:54:35,601] [INFO] [timer.py:197:stop] 0/2048, RunningAvgSamplesPerSec=5.863603788022735, CurrSamplesPerSec=5.295188185541611, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1012/5000 [3:32:11<21:32:54, 19.45s/it][2022-12-20 13:54:46,847] [INFO] [timer.py:197:stop] 0/2050, RunningAvgSamplesPerSec=5.863493825304255, CurrSamplesPerSec=5.365005978273556, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1013/5000 [3:32:22<18:49:00, 16.99s/it][2022-12-20 13:54:57,865] [INFO] [timer.py:197:stop] 0/2052, RunningAvgSamplesPerSec=5.863507377947548, CurrSamplesPerSec=5.347278298037127, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1014/5000 [3:32:33<16:49:40, 15.20s/it][2022-12-20 13:55:09,105] [INFO] [timer.py:197:stop] 0/2054, RunningAvgSamplesPerSec=5.863401559240622, CurrSamplesPerSec=5.156595057476339, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1015/5000 [3:32:44<15:30:33, 14.01s/it][2022-12-20 13:55:20,086] [INFO] [timer.py:197:stop] 0/2056, RunningAvgSamplesPerSec=5.8634313586809395, CurrSamplesPerSec=5.3284888337248315, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1016/5000 [3:32:55<14:29:58, 13.10s/it][2022-12-20 13:55:31,107] [INFO] [timer.py:197:stop] 0/2058, RunningAvgSamplesPerSec=5.863441457895085, CurrSamplesPerSec=5.310022328424466, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1017/5000 [3:33:06<13:48:18, 12.48s/it][2022-12-20 13:55:42,249] [INFO] [logging.py:68:log_dist] [Rank 0] step=1030, skipped=3, lr=[8.831111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:55:42,251] [INFO] [timer.py:197:stop] 0/2060, RunningAvgSamplesPerSec=5.863389686432189, CurrSamplesPerSec=5.313772575948109, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1018/5000 [3:33:17<13:21:32, 12.08s/it][2022-12-20 13:55:53,267] [INFO] [timer.py:197:stop] 0/2062, RunningAvgSamplesPerSec=5.863401051275269, CurrSamplesPerSec=5.31035070233564, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1019/5000 [3:33:28<13:00:13, 11.76s/it][2022-12-20 13:56:04,405] [INFO] [timer.py:197:stop] 0/2064, RunningAvgSamplesPerSec=5.863349304991636, CurrSamplesPerSec=5.2426029202440985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1020/5000 [3:33:40<12:47:39, 11.57s/it][2022-12-20 13:56:15,416] [INFO] [timer.py:197:stop] 0/2066, RunningAvgSamplesPerSec=5.863363634420066, CurrSamplesPerSec=5.31510353697576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1021/5000 [3:33:51<12:36:17, 11.40s/it][2022-12-20 13:56:26,417] [INFO] [timer.py:197:stop] 0/2068, RunningAvgSamplesPerSec=5.863382961010374, CurrSamplesPerSec=5.351581917232091, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1022/5000 [3:34:02<12:28:04, 11.28s/it][2022-12-20 13:56:37,441] [INFO] [timer.py:197:stop] 0/2070, RunningAvgSamplesPerSec=5.863399234920631, CurrSamplesPerSec=5.350668165134254, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1023/5000 [3:34:13<12:22:44, 11.21s/it][2022-12-20 13:56:48,532] [INFO] [timer.py:197:stop] 0/2072, RunningAvgSamplesPerSec=5.863372749033641, CurrSamplesPerSec=5.263555079867624, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1024/5000 [3:34:24<12:20:16, 11.17s/it][2022-12-20 13:56:59,576] [INFO] [timer.py:197:stop] 0/2074, RunningAvgSamplesPerSec=5.86337001283808, CurrSamplesPerSec=5.323514610363855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 20%|██        | 1025/5000 [3:34:35<12:17:35, 11.13s/it]
                                                        
{'loss': 0.003, 'learning_rate': 8.815555555555557e-06, 'epoch': 24.99}

 20%|██        | 1025/5000 [3:34:35<12:17:35, 11.13s/it][2022-12-20 13:57:09,695] [INFO] [timer.py:197:stop] 0/2076, RunningAvgSamplesPerSec=5.863850987293994, CurrSamplesPerSec=5.35015435667869, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1026/5000 [3:34:50<13:35:09, 12.31s/it][2022-12-20 13:57:20,677] [INFO] [timer.py:197:stop] 0/2078, RunningAvgSamplesPerSec=5.863879770832041, CurrSamplesPerSec=5.3430524234289845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1027/5000 [3:35:01<13:08:33, 11.91s/it][2022-12-20 13:57:31,743] [INFO] [logging.py:68:log_dist] [Rank 0] step=1040, skipped=3, lr=[8.80888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:57:31,745] [INFO] [timer.py:197:stop] 0/2080, RunningAvgSamplesPerSec=5.863862710020031, CurrSamplesPerSec=5.26310946021707, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1028/5000 [3:35:12<12:51:43, 11.66s/it][2022-12-20 13:57:42,728] [INFO] [timer.py:197:stop] 0/2082, RunningAvgSamplesPerSec=5.863890630642203, CurrSamplesPerSec=5.341398336648782, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1029/5000 [3:35:23<12:38:04, 11.45s/it][2022-12-20 13:57:53,734] [INFO] [timer.py:197:stop] 0/2084, RunningAvgSamplesPerSec=5.863905922714832, CurrSamplesPerSec=5.3176779175110465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1030/5000 [3:35:34<12:31:24, 11.36s/it][2022-12-20 13:58:04,901] [INFO] [timer.py:197:stop] 0/2086, RunningAvgSamplesPerSec=5.863839121847008, CurrSamplesPerSec=5.284583621939008, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1031/5000 [3:35:45<12:24:15, 11.25s/it][2022-12-20 13:58:15,859] [INFO] [timer.py:197:stop] 0/2088, RunningAvgSamplesPerSec=5.863879072806387, CurrSamplesPerSec=5.3251217206869645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1032/5000 [3:35:56<12:18:55, 11.17s/it][2022-12-20 13:58:26,836] [INFO] [timer.py:197:stop] 0/2090, RunningAvgSamplesPerSec=5.8639157452497646, CurrSamplesPerSec=5.338732988467036, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1033/5000 [3:36:07<12:14:49, 11.11s/it][2022-12-20 13:58:37,823] [INFO] [timer.py:197:stop] 0/2092, RunningAvgSamplesPerSec=5.863947036883069, CurrSamplesPerSec=5.3289015856867294, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1034/5000 [3:36:18<12:11:44, 11.07s/it][2022-12-20 13:58:48,828] [INFO] [timer.py:197:stop] 0/2094, RunningAvgSamplesPerSec=5.863962377593371, CurrSamplesPerSec=5.295487566144904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1035/5000 [3:36:29<12:11:05, 11.06s/it][2022-12-20 13:58:59,831] [INFO] [timer.py:197:stop] 0/2096, RunningAvgSamplesPerSec=5.863983407890965, CurrSamplesPerSec=5.341273136407529, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1036/5000 [3:36:40<12:10:13, 11.05s/it][2022-12-20 13:59:10,881] [INFO] [timer.py:197:stop] 0/2098, RunningAvgSamplesPerSec=5.86397699376101, CurrSamplesPerSec=5.316498131113512, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1037/5000 [3:36:51<12:10:17, 11.06s/it][2022-12-20 13:59:21,953] [INFO] [logging.py:68:log_dist] [Rank 0] step=1050, skipped=3, lr=[8.786666666666668e-06], mom=[[0.9, 0.999]]
[2022-12-20 13:59:21,955] [INFO] [timer.py:197:stop] 0/2100, RunningAvgSamplesPerSec=5.8639581315535265, CurrSamplesPerSec=5.3083583934251495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1038/5000 [3:37:02<12:10:17, 11.06s/it][2022-12-20 13:59:32,979] [INFO] [timer.py:197:stop] 0/2102, RunningAvgSamplesPerSec=5.863964950532257, CurrSamplesPerSec=5.346437784905685, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1039/5000 [3:37:13<12:08:54, 11.04s/it][2022-12-20 13:59:43,980] [INFO] [timer.py:197:stop] 0/2104, RunningAvgSamplesPerSec=5.863983120647728, CurrSamplesPerSec=5.343611459441752, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1040/5000 [3:37:24<12:08:23, 11.04s/it][2022-12-20 13:59:55,002] [INFO] [timer.py:197:stop] 0/2106, RunningAvgSamplesPerSec=5.863990451893773, CurrSamplesPerSec=5.345059796031264, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1041/5000 [3:37:35<12:07:24, 11.02s/it][2022-12-20 14:00:06,022] [INFO] [timer.py:197:stop] 0/2108, RunningAvgSamplesPerSec=5.863998780648715, CurrSamplesPerSec=5.324498108366836, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1042/5000 [3:37:46<12:07:28, 11.03s/it][2022-12-20 14:00:17,072] [INFO] [timer.py:197:stop] 0/2110, RunningAvgSamplesPerSec=5.863999195673888, CurrSamplesPerSec=5.312305608950852, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1043/5000 [3:37:57<12:07:37, 11.03s/it][2022-12-20 14:00:28,106] [INFO] [timer.py:197:stop] 0/2112, RunningAvgSamplesPerSec=5.864001645544187, CurrSamplesPerSec=5.323496662842479, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1044/5000 [3:38:08<12:07:26, 11.03s/it][2022-12-20 14:00:39,139] [INFO] [timer.py:197:stop] 0/2114, RunningAvgSamplesPerSec=5.864003600212431, CurrSamplesPerSec=5.323034925284297, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1045/5000 [3:38:19<12:07:06, 11.03s/it][2022-12-20 14:00:50,130] [INFO] [timer.py:197:stop] 0/2116, RunningAvgSamplesPerSec=5.864035291846092, CurrSamplesPerSec=5.353817797026638, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1046/5000 [3:38:30<12:06:05, 11.02s/it][2022-12-20 14:01:01,168] [INFO] [timer.py:197:stop] 0/2118, RunningAvgSamplesPerSec=5.864033742378036, CurrSamplesPerSec=5.308203456734344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1047/5000 [3:38:41<12:06:21, 11.02s/it][2022-12-20 14:01:12,200] [INFO] [logging.py:68:log_dist] [Rank 0] step=1060, skipped=3, lr=[8.764444444444446e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:01:12,201] [INFO] [timer.py:197:stop] 0/2120, RunningAvgSamplesPerSec=5.864035287320744, CurrSamplesPerSec=5.315322236073764, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1048/5000 [3:38:52<12:05:25, 11.01s/it][2022-12-20 14:01:23,199] [INFO] [timer.py:197:stop] 0/2122, RunningAvgSamplesPerSec=5.86405479210543, CurrSamplesPerSec=5.30661325434298, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1049/5000 [3:39:03<12:05:25, 11.02s/it][2022-12-20 14:01:34,236] [INFO] [timer.py:197:stop] 0/2124, RunningAvgSamplesPerSec=5.864053769168871, CurrSamplesPerSec=5.293740647201711, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1050/5000 [3:39:14<12:06:17, 11.03s/it]
                                                        
{'loss': 0.0027, 'learning_rate': 8.76e-06, 'epoch': 25.6}

 21%|██        | 1050/5000 [3:39:14<12:06:17, 11.03s/it][2022-12-20 14:01:45,307] [INFO] [timer.py:197:stop] 0/2126, RunningAvgSamplesPerSec=5.8640368067427655, CurrSamplesPerSec=5.293431651952121, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1051/5000 [3:39:25<12:06:19, 11.04s/it][2022-12-20 14:01:56,339] [INFO] [timer.py:197:stop] 0/2128, RunningAvgSamplesPerSec=5.864039055321194, CurrSamplesPerSec=5.3028689117070025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1052/5000 [3:39:36<12:06:32, 11.04s/it][2022-12-20 14:02:07,354] [INFO] [timer.py:197:stop] 0/2130, RunningAvgSamplesPerSec=5.864049667097402, CurrSamplesPerSec=5.338703258656914, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1053/5000 [3:39:47<12:06:02, 11.04s/it][2022-12-20 14:02:18,399] [INFO] [timer.py:197:stop] 0/2132, RunningAvgSamplesPerSec=5.86404652139603, CurrSamplesPerSec=5.321801484483407, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1054/5000 [3:39:58<12:05:51, 11.04s/it][2022-12-20 14:02:29,475] [INFO] [timer.py:197:stop] 0/2134, RunningAvgSamplesPerSec=5.864026025930713, CurrSamplesPerSec=5.286453177309822, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1055/5000 [3:40:10<12:06:35, 11.05s/it][2022-12-20 14:02:40,507] [INFO] [timer.py:197:stop] 0/2136, RunningAvgSamplesPerSec=5.864028488316095, CurrSamplesPerSec=5.332904214420549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1056/5000 [3:40:21<12:06:04, 11.05s/it][2022-12-20 14:02:51,536] [INFO] [timer.py:197:stop] 0/2138, RunningAvgSamplesPerSec=5.864031985893871, CurrSamplesPerSec=5.337082211265904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1057/5000 [3:40:32<12:05:24, 11.04s/it][2022-12-20 14:03:02,586] [INFO] [logging.py:68:log_dist] [Rank 0] step=1070, skipped=3, lr=[8.742222222222224e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:03:02,587] [INFO] [timer.py:197:stop] 0/2140, RunningAvgSamplesPerSec=5.8640244493423435, CurrSamplesPerSec=5.310837560404776, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1058/5000 [3:40:43<12:05:24, 11.04s/it][2022-12-20 14:03:13,604] [INFO] [timer.py:197:stop] 0/2142, RunningAvgSamplesPerSec=5.864034272343851, CurrSamplesPerSec=5.3383492870029, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1059/5000 [3:40:54<12:03:53, 11.02s/it][2022-12-20 14:03:24,612] [INFO] [timer.py:197:stop] 0/2144, RunningAvgSamplesPerSec=5.864048094117084, CurrSamplesPerSec=5.307198897070904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1060/5000 [3:41:05<12:03:30, 11.02s/it][2022-12-20 14:03:35,591] [INFO] [timer.py:197:stop] 0/2146, RunningAvgSamplesPerSec=5.864076728453261, CurrSamplesPerSec=5.335307109721164, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1061/5000 [3:41:16<12:02:47, 11.01s/it][2022-12-20 14:03:46,585] [INFO] [timer.py:197:stop] 0/2148, RunningAvgSamplesPerSec=5.864098346108194, CurrSamplesPerSec=5.333846247510869, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██        | 1062/5000 [3:41:27<12:02:51, 11.01s/it][2022-12-20 14:03:57,651] [INFO] [timer.py:197:stop] 0/2150, RunningAvgSamplesPerSec=5.864083366518799, CurrSamplesPerSec=5.294184577836855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1063/5000 [3:41:38<12:03:33, 11.03s/it][2022-12-20 14:04:08,630] [INFO] [timer.py:197:stop] 0/2152, RunningAvgSamplesPerSec=5.864111529783501, CurrSamplesPerSec=5.364261074716845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1064/5000 [3:41:49<12:02:03, 11.01s/it][2022-12-20 14:04:19,619] [INFO] [timer.py:197:stop] 0/2154, RunningAvgSamplesPerSec=5.8641352790665415, CurrSamplesPerSec=5.339265845281512, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1065/5000 [3:42:00<12:02:10, 11.01s/it][2022-12-20 14:04:30,722] [INFO] [timer.py:197:stop] 0/2156, RunningAvgSamplesPerSec=5.864102400751884, CurrSamplesPerSec=5.26852702491582, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1066/5000 [3:42:11<12:03:32, 11.04s/it][2022-12-20 14:04:40,849] [INFO] [timer.py:197:stop] 0/2158, RunningAvgSamplesPerSec=5.864553343976343, CurrSamplesPerSec=6.2492822306413425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 14:04:51,835] [INFO] [logging.py:68:log_dist] [Rank 0] step=1080, skipped=3, lr=[8.720000000000001e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:04:51,836] [INFO] [timer.py:197:stop] 0/2160, RunningAvgSamplesPerSec=5.864581100801598, CurrSamplesPerSec=5.324717792519189, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1067/5000 [3:42:27<13:44:09, 12.57s/it][2022-12-20 14:05:02,847] [INFO] [timer.py:197:stop] 0/2162, RunningAvgSamplesPerSec=5.8645934464406455, CurrSamplesPerSec=5.324314348048252, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1068/5000 [3:42:38<13:13:14, 12.10s/it][2022-12-20 14:05:13,903] [INFO] [timer.py:197:stop] 0/2164, RunningAvgSamplesPerSec=5.864582772100132, CurrSamplesPerSec=5.298026662843538, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1069/5000 [3:42:49<12:52:26, 11.79s/it][2022-12-20 14:05:24,931] [INFO] [timer.py:197:stop] 0/2166, RunningAvgSamplesPerSec=5.864592894167747, CurrSamplesPerSec=5.314301092839466, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1070/5000 [3:43:00<12:37:15, 11.56s/it][2022-12-20 14:05:35,942] [INFO] [timer.py:197:stop] 0/2168, RunningAvgSamplesPerSec=5.86460474887039, CurrSamplesPerSec=5.347219500383081, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1071/5000 [3:43:11<12:26:15, 11.40s/it][2022-12-20 14:05:46,910] [INFO] [timer.py:197:stop] 0/2170, RunningAvgSamplesPerSec=5.864638793598877, CurrSamplesPerSec=5.353111438016366, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1072/5000 [3:43:22<12:17:40, 11.27s/it][2022-12-20 14:05:57,981] [INFO] [timer.py:197:stop] 0/2172, RunningAvgSamplesPerSec=5.864621245502965, CurrSamplesPerSec=5.26190260007154, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1073/5000 [3:43:33<12:13:36, 11.21s/it][2022-12-20 14:06:09,027] [INFO] [timer.py:197:stop] 0/2174, RunningAvgSamplesPerSec=5.864618818421614, CurrSamplesPerSec=5.308806669068894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 21%|██▏       | 1074/5000 [3:43:44<12:10:13, 11.16s/it][2022-12-20 14:06:20,007] [INFO] [timer.py:197:stop] 0/2176, RunningAvgSamplesPerSec=5.864646519507423, CurrSamplesPerSec=5.366300724234963, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1075/5000 [3:43:55<12:06:30, 11.11s/it]
                                                        
{'loss': 0.0025, 'learning_rate': 8.702222222222222e-06, 'epoch': 26.22}

 22%|██▏       | 1075/5000 [3:43:55<12:06:30, 11.11s/it][2022-12-20 14:06:31,031] [INFO] [timer.py:197:stop] 0/2178, RunningAvgSamplesPerSec=5.864653092648156, CurrSamplesPerSec=5.3329116307042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1076/5000 [3:44:06<12:04:43, 11.08s/it][2022-12-20 14:06:42,043] [INFO] [logging.py:68:log_dist] [Rank 0] step=1090, skipped=3, lr=[8.697777777777779e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:06:42,045] [INFO] [timer.py:197:stop] 0/2180, RunningAvgSamplesPerSec=5.8646634641746385, CurrSamplesPerSec=5.34102742939782, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1077/5000 [3:44:17<12:03:13, 11.06s/it][2022-12-20 14:06:52,978] [INFO] [timer.py:197:stop] 0/2182, RunningAvgSamplesPerSec=5.864714493048281, CurrSamplesPerSec=5.341413429094641, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1078/5000 [3:44:28<12:00:31, 11.02s/it][2022-12-20 14:07:03,954] [INFO] [timer.py:197:stop] 0/2184, RunningAvgSamplesPerSec=5.864743401605117, CurrSamplesPerSec=5.343175367281848, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1079/5000 [3:44:39<11:59:25, 11.01s/it][2022-12-20 14:07:15,044] [INFO] [timer.py:197:stop] 0/2186, RunningAvgSamplesPerSec=5.864716731782743, CurrSamplesPerSec=5.299642901789582, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1080/5000 [3:44:50<12:00:49, 11.03s/it][2022-12-20 14:07:26,099] [INFO] [timer.py:197:stop] 0/2188, RunningAvgSamplesPerSec=5.864706815875364, CurrSamplesPerSec=5.3000855207086515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1081/5000 [3:45:01<12:01:04, 11.04s/it][2022-12-20 14:07:37,147] [INFO] [timer.py:197:stop] 0/2190, RunningAvgSamplesPerSec=5.864708923970429, CurrSamplesPerSec=5.299555642386944, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1082/5000 [3:45:12<12:01:03, 11.04s/it][2022-12-20 14:07:48,140] [INFO] [timer.py:197:stop] 0/2192, RunningAvgSamplesPerSec=5.864729893569766, CurrSamplesPerSec=5.347439998253347, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1083/5000 [3:45:23<11:59:54, 11.03s/it][2022-12-20 14:07:59,135] [INFO] [timer.py:197:stop] 0/2194, RunningAvgSamplesPerSec=5.864750009143748, CurrSamplesPerSec=5.315612950608003, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1084/5000 [3:45:34<11:59:05, 11.02s/it][2022-12-20 14:08:10,108] [INFO] [timer.py:197:stop] 0/2196, RunningAvgSamplesPerSec=5.864780426585327, CurrSamplesPerSec=5.364609056590229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1085/5000 [3:45:45<11:58:02, 11.00s/it][2022-12-20 14:08:21,117] [INFO] [timer.py:197:stop] 0/2198, RunningAvgSamplesPerSec=5.864795321272736, CurrSamplesPerSec=5.3543314550037655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1086/5000 [3:45:56<11:57:56, 11.01s/it][2022-12-20 14:08:32,168] [INFO] [logging.py:68:log_dist] [Rank 0] step=1100, skipped=3, lr=[8.675555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:08:32,170] [INFO] [timer.py:197:stop] 0/2200, RunningAvgSamplesPerSec=5.864794535738305, CurrSamplesPerSec=5.298841771703382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1087/5000 [3:46:07<11:58:40, 11.02s/it][2022-12-20 14:08:43,221] [INFO] [timer.py:197:stop] 0/2202, RunningAvgSamplesPerSec=5.8647877379434545, CurrSamplesPerSec=5.320138806364519, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1088/5000 [3:46:18<11:59:06, 11.03s/it][2022-12-20 14:08:54,229] [INFO] [timer.py:197:stop] 0/2204, RunningAvgSamplesPerSec=5.8648068275083585, CurrSamplesPerSec=5.35050648292483, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1089/5000 [3:46:29<11:58:30, 11.02s/it][2022-12-20 14:09:05,311] [INFO] [timer.py:197:stop] 0/2206, RunningAvgSamplesPerSec=5.86478458568544, CurrSamplesPerSec=5.29473990915237, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1090/5000 [3:46:40<11:59:28, 11.04s/it][2022-12-20 14:09:16,335] [INFO] [timer.py:197:stop] 0/2208, RunningAvgSamplesPerSec=5.864790337118027, CurrSamplesPerSec=5.3328898056998515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1091/5000 [3:46:51<11:58:58, 11.04s/it][2022-12-20 14:09:27,393] [INFO] [timer.py:197:stop] 0/2210, RunningAvgSamplesPerSec=5.864779369409788, CurrSamplesPerSec=5.32268535012933, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1092/5000 [3:47:02<11:59:13, 11.04s/it][2022-12-20 14:09:38,548] [INFO] [timer.py:197:stop] 0/2212, RunningAvgSamplesPerSec=5.864721783778854, CurrSamplesPerSec=5.230506379030451, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1093/5000 [3:47:14<12:01:14, 11.08s/it][2022-12-20 14:09:49,614] [INFO] [timer.py:197:stop] 0/2214, RunningAvgSamplesPerSec=5.864709157834904, CurrSamplesPerSec=5.301934436927559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1094/5000 [3:47:25<12:00:50, 11.07s/it][2022-12-20 14:10:00,570] [INFO] [timer.py:197:stop] 0/2216, RunningAvgSamplesPerSec=5.864747393800292, CurrSamplesPerSec=5.346892941296639, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1095/5000 [3:47:36<11:58:22, 11.04s/it][2022-12-20 14:10:11,628] [INFO] [timer.py:197:stop] 0/2218, RunningAvgSamplesPerSec=5.8647376400973705, CurrSamplesPerSec=5.303320451628331, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1096/5000 [3:47:47<11:58:35, 11.04s/it][2022-12-20 14:10:22,671] [INFO] [logging.py:68:log_dist] [Rank 0] step=1110, skipped=3, lr=[8.653333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:10:22,673] [INFO] [timer.py:197:stop] 0/2220, RunningAvgSamplesPerSec=5.864739129465555, CurrSamplesPerSec=5.321221264343892, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1097/5000 [3:47:58<11:58:25, 11.04s/it][2022-12-20 14:10:33,716] [INFO] [timer.py:197:stop] 0/2222, RunningAvgSamplesPerSec=5.864735069539521, CurrSamplesPerSec=5.3211984800985315, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1098/5000 [3:48:09<11:58:13, 11.04s/it][2022-12-20 14:10:44,741] [INFO] [timer.py:197:stop] 0/2224, RunningAvgSamplesPerSec=5.8647404409024055, CurrSamplesPerSec=5.306076406342588, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1099/5000 [3:48:20<11:57:40, 11.04s/it][2022-12-20 14:10:55,794] [INFO] [timer.py:197:stop] 0/2226, RunningAvgSamplesPerSec=5.8647349478035204, CurrSamplesPerSec=5.307838614455876, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1100/5000 [3:48:31<11:57:46, 11.04s/it]
                                                        
{'loss': 0.0019, 'learning_rate': 8.646666666666668e-06, 'epoch': 26.82}

 22%|██▏       | 1100/5000 [3:48:31<11:57:46, 11.04s/it][2022-12-20 14:11:06,859] [INFO] [timer.py:197:stop] 0/2228, RunningAvgSamplesPerSec=5.864721885493288, CurrSamplesPerSec=5.291202735734523, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1101/5000 [3:48:42<11:58:00, 11.05s/it][2022-12-20 14:11:17,914] [INFO] [timer.py:197:stop] 0/2230, RunningAvgSamplesPerSec=5.864716545123169, CurrSamplesPerSec=5.302519885280004, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1102/5000 [3:48:53<11:57:57, 11.05s/it][2022-12-20 14:11:28,939] [INFO] [timer.py:197:stop] 0/2232, RunningAvgSamplesPerSec=5.864721719243963, CurrSamplesPerSec=5.308745564747245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1103/5000 [3:49:04<11:57:15, 11.04s/it][2022-12-20 14:11:39,973] [INFO] [timer.py:197:stop] 0/2234, RunningAvgSamplesPerSec=5.864722708580744, CurrSamplesPerSec=5.330234633577298, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1104/5000 [3:49:15<11:56:53, 11.04s/it][2022-12-20 14:11:51,035] [INFO] [timer.py:197:stop] 0/2236, RunningAvgSamplesPerSec=5.8647136474348835, CurrSamplesPerSec=5.312328737628104, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1105/5000 [3:49:26<11:57:07, 11.05s/it][2022-12-20 14:12:02,034] [INFO] [timer.py:197:stop] 0/2238, RunningAvgSamplesPerSec=5.864733430074749, CurrSamplesPerSec=5.332309707474836, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1106/5000 [3:49:37<11:56:00, 11.03s/it][2022-12-20 14:12:13,051] [INFO] [logging.py:68:log_dist] [Rank 0] step=1120, skipped=3, lr=[8.631111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:12:13,052] [INFO] [timer.py:197:stop] 0/2240, RunningAvgSamplesPerSec=5.864743512827763, CurrSamplesPerSec=5.31612983157818, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1107/5000 [3:49:48<11:55:32, 11.03s/it][2022-12-20 14:12:23,219] [INFO] [timer.py:197:stop] 0/2242, RunningAvgSamplesPerSec=5.86515615693459, CurrSamplesPerSec=5.304526890794296, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1108/5000 [3:50:03<13:14:29, 12.25s/it][2022-12-20 14:12:34,224] [INFO] [timer.py:197:stop] 0/2244, RunningAvgSamplesPerSec=5.865170598574616, CurrSamplesPerSec=5.321723622140125, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1109/5000 [3:50:14<12:50:31, 11.88s/it][2022-12-20 14:12:45,264] [INFO] [timer.py:197:stop] 0/2246, RunningAvgSamplesPerSec=5.865168219554229, CurrSamplesPerSec=5.3102681323286465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1110/5000 [3:50:25<12:34:13, 11.63s/it][2022-12-20 14:12:56,290] [INFO] [timer.py:197:stop] 0/2248, RunningAvgSamplesPerSec=5.865173781563321, CurrSamplesPerSec=5.336660764378463, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1111/5000 [3:50:36<12:21:27, 11.44s/it][2022-12-20 14:13:07,316] [INFO] [timer.py:197:stop] 0/2250, RunningAvgSamplesPerSec=5.865180007392161, CurrSamplesPerSec=5.30172772809063, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1112/5000 [3:50:47<12:13:36, 11.32s/it][2022-12-20 14:13:18,365] [INFO] [timer.py:197:stop] 0/2252, RunningAvgSamplesPerSec=5.865172791576657, CurrSamplesPerSec=5.295400234600223, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1113/5000 [3:50:58<12:07:25, 11.23s/it][2022-12-20 14:13:29,488] [INFO] [timer.py:197:stop] 0/2254, RunningAvgSamplesPerSec=5.865130687223691, CurrSamplesPerSec=5.2008883121909495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1114/5000 [3:51:10<12:06:29, 11.22s/it][2022-12-20 14:13:40,522] [INFO] [timer.py:197:stop] 0/2256, RunningAvgSamplesPerSec=5.8651382310160844, CurrSamplesPerSec=5.336662674108486, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1115/5000 [3:51:21<12:01:37, 11.14s/it][2022-12-20 14:13:51,582] [INFO] [timer.py:197:stop] 0/2258, RunningAvgSamplesPerSec=5.865126760247001, CurrSamplesPerSec=5.263164565181997, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1116/5000 [3:51:32<11:59:57, 11.12s/it][2022-12-20 14:14:02,552] [INFO] [logging.py:68:log_dist] [Rank 0] step=1130, skipped=3, lr=[8.60888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:14:02,554] [INFO] [timer.py:197:stop] 0/2260, RunningAvgSamplesPerSec=5.865157653267203, CurrSamplesPerSec=5.349653653685572, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1117/5000 [3:51:43<11:57:41, 11.09s/it][2022-12-20 14:14:13,565] [INFO] [timer.py:197:stop] 0/2262, RunningAvgSamplesPerSec=5.865168645626259, CurrSamplesPerSec=5.350817911199397, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1118/5000 [3:51:54<11:54:42, 11.05s/it][2022-12-20 14:14:24,525] [INFO] [timer.py:197:stop] 0/2264, RunningAvgSamplesPerSec=5.865204938025142, CurrSamplesPerSec=5.339099116989073, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1119/5000 [3:52:05<11:54:21, 11.04s/it][2022-12-20 14:14:35,609] [INFO] [timer.py:197:stop] 0/2266, RunningAvgSamplesPerSec=5.865184046889684, CurrSamplesPerSec=5.297849116581928, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1120/5000 [3:52:16<11:53:31, 11.03s/it][2022-12-20 14:14:46,596] [INFO] [timer.py:197:stop] 0/2268, RunningAvgSamplesPerSec=5.865206825467563, CurrSamplesPerSec=5.318219644525164, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1121/5000 [3:52:27<11:51:57, 11.01s/it][2022-12-20 14:14:57,525] [INFO] [timer.py:197:stop] 0/2270, RunningAvgSamplesPerSec=5.8652567208296915, CurrSamplesPerSec=5.347790701890485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1122/5000 [3:52:38<11:50:39, 11.00s/it][2022-12-20 14:15:08,487] [INFO] [timer.py:197:stop] 0/2272, RunningAvgSamplesPerSec=5.86529128143706, CurrSamplesPerSec=5.3418541236015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1123/5000 [3:52:49<11:50:13, 10.99s/it][2022-12-20 14:15:19,511] [INFO] [timer.py:197:stop] 0/2274, RunningAvgSamplesPerSec=5.865295920018038, CurrSamplesPerSec=5.304192319310985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▏       | 1124/5000 [3:53:00<11:51:11, 11.01s/it][2022-12-20 14:15:30,544] [INFO] [timer.py:197:stop] 0/2276, RunningAvgSamplesPerSec=5.865296439071898, CurrSamplesPerSec=5.3206972760597475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 22%|██▎       | 1125/5000 [3:53:11<11:51:39, 11.02s/it]
                                                        
{'loss': 0.0021, 'learning_rate': 8.591111111111112e-06, 'epoch': 27.43}

 22%|██▎       | 1125/5000 [3:53:11<11:51:39, 11.02s/it][2022-12-20 14:15:41,579] [INFO] [timer.py:197:stop] 0/2278, RunningAvgSamplesPerSec=5.865298541262584, CurrSamplesPerSec=5.327499843212781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1126/5000 [3:53:22<11:51:17, 11.02s/it][2022-12-20 14:15:52,588] [INFO] [logging.py:68:log_dist] [Rank 0] step=1140, skipped=3, lr=[8.586666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:15:52,590] [INFO] [timer.py:197:stop] 0/2280, RunningAvgSamplesPerSec=5.865309157514421, CurrSamplesPerSec=5.325291591630112, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1127/5000 [3:53:33<11:51:22, 11.02s/it][2022-12-20 14:16:03,581] [INFO] [timer.py:197:stop] 0/2282, RunningAvgSamplesPerSec=5.86532916902633, CurrSamplesPerSec=5.361328714891081, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1128/5000 [3:53:44<11:49:32, 11.00s/it][2022-12-20 14:16:14,516] [INFO] [timer.py:197:stop] 0/2284, RunningAvgSamplesPerSec=5.865375576271883, CurrSamplesPerSec=5.361487410916991, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1129/5000 [3:53:55<11:48:27, 10.98s/it][2022-12-20 14:16:25,488] [INFO] [timer.py:197:stop] 0/2286, RunningAvgSamplesPerSec=5.865404946836134, CurrSamplesPerSec=5.341338392846767, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1130/5000 [3:54:06<11:48:17, 10.98s/it][2022-12-20 14:16:36,485] [INFO] [timer.py:197:stop] 0/2288, RunningAvgSamplesPerSec=5.86542233096877, CurrSamplesPerSec=5.327405531815613, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1131/5000 [3:54:17<11:49:01, 11.00s/it][2022-12-20 14:16:47,556] [INFO] [timer.py:197:stop] 0/2290, RunningAvgSamplesPerSec=5.865404769565402, CurrSamplesPerSec=5.288957355158159, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1132/5000 [3:54:28<11:50:04, 11.01s/it][2022-12-20 14:16:58,583] [INFO] [timer.py:197:stop] 0/2292, RunningAvgSamplesPerSec=5.86540775171255, CurrSamplesPerSec=5.317277645627119, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1133/5000 [3:54:39<11:49:47, 11.01s/it][2022-12-20 14:17:09,595] [INFO] [timer.py:197:stop] 0/2294, RunningAvgSamplesPerSec=5.865417214120428, CurrSamplesPerSec=5.314231024643213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1134/5000 [3:54:50<11:49:50, 11.02s/it][2022-12-20 14:17:20,620] [INFO] [timer.py:197:stop] 0/2296, RunningAvgSamplesPerSec=5.865421892145623, CurrSamplesPerSec=5.316322503357713, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1135/5000 [3:55:01<11:49:14, 11.01s/it][2022-12-20 14:17:31,627] [INFO] [timer.py:197:stop] 0/2298, RunningAvgSamplesPerSec=5.865434501497301, CurrSamplesPerSec=5.3049684388164575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1136/5000 [3:55:12<11:48:50, 11.01s/it][2022-12-20 14:17:42,609] [INFO] [logging.py:68:log_dist] [Rank 0] step=1150, skipped=3, lr=[8.564444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:17:42,611] [INFO] [timer.py:197:stop] 0/2300, RunningAvgSamplesPerSec=5.865457691582508, CurrSamplesPerSec=5.318497820298302, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1137/5000 [3:55:23<11:48:56, 11.01s/it][2022-12-20 14:17:53,605] [INFO] [timer.py:197:stop] 0/2302, RunningAvgSamplesPerSec=5.865475697674752, CurrSamplesPerSec=5.34173039016067, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1138/5000 [3:55:34<11:47:50, 11.00s/it][2022-12-20 14:18:04,601] [INFO] [timer.py:197:stop] 0/2304, RunningAvgSamplesPerSec=5.8655005303686965, CurrSamplesPerSec=5.326890684296372, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1139/5000 [3:55:45<11:48:30, 11.01s/it][2022-12-20 14:18:15,643] [INFO] [timer.py:197:stop] 0/2306, RunningAvgSamplesPerSec=5.865496280394534, CurrSamplesPerSec=5.313038675712788, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1140/5000 [3:55:56<11:48:59, 11.02s/it][2022-12-20 14:18:26,673] [INFO] [timer.py:197:stop] 0/2308, RunningAvgSamplesPerSec=5.865497587354025, CurrSamplesPerSec=5.325734067573845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1141/5000 [3:56:07<11:49:07, 11.03s/it][2022-12-20 14:18:37,709] [INFO] [timer.py:197:stop] 0/2310, RunningAvgSamplesPerSec=5.865497024436438, CurrSamplesPerSec=5.327805426617686, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1142/5000 [3:56:18<11:48:38, 11.02s/it][2022-12-20 14:18:48,735] [INFO] [timer.py:197:stop] 0/2312, RunningAvgSamplesPerSec=5.865500428844639, CurrSamplesPerSec=5.31282205719024, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1143/5000 [3:56:29<11:48:14, 11.02s/it][2022-12-20 14:18:59,761] [INFO] [timer.py:197:stop] 0/2314, RunningAvgSamplesPerSec=5.8655046402636675, CurrSamplesPerSec=5.299324010860995, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1144/5000 [3:56:40<11:48:13, 11.02s/it][2022-12-20 14:19:10,784] [INFO] [timer.py:197:stop] 0/2316, RunningAvgSamplesPerSec=5.865509465851486, CurrSamplesPerSec=5.301385552535815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1145/5000 [3:56:51<11:47:47, 11.02s/it][2022-12-20 14:19:21,767] [INFO] [timer.py:197:stop] 0/2318, RunningAvgSamplesPerSec=5.865533465552191, CurrSamplesPerSec=5.3235148215118855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1146/5000 [3:57:02<11:47:26, 11.01s/it][2022-12-20 14:19:32,804] [INFO] [logging.py:68:log_dist] [Rank 0] step=1160, skipped=3, lr=[8.542222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:19:32,805] [INFO] [timer.py:197:stop] 0/2320, RunningAvgSamplesPerSec=5.865532716517903, CurrSamplesPerSec=5.295595167583055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1147/5000 [3:57:13<11:48:10, 11.03s/it][2022-12-20 14:19:43,844] [INFO] [timer.py:197:stop] 0/2322, RunningAvgSamplesPerSec=5.865530540159725, CurrSamplesPerSec=5.315744319380331, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1148/5000 [3:57:24<11:47:35, 11.02s/it][2022-12-20 14:19:53,950] [INFO] [timer.py:197:stop] 0/2324, RunningAvgSamplesPerSec=5.865956033698763, CurrSamplesPerSec=6.24039639415963, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 14:20:05,028] [INFO] [timer.py:197:stop] 0/2326, RunningAvgSamplesPerSec=5.865934394399958, CurrSamplesPerSec=5.289685451108795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1149/5000 [3:57:40<13:28:16, 12.59s/it][2022-12-20 14:20:16,051] [INFO] [timer.py:197:stop] 0/2328, RunningAvgSamplesPerSec=5.865939758476308, CurrSamplesPerSec=5.307168678016946, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1150/5000 [3:57:51<12:57:50, 12.12s/it]
                                                        
{'loss': 0.0018, 'learning_rate': 8.533333333333335e-06, 'epoch': 28.05}

 23%|██▎       | 1150/5000 [3:57:51<12:57:50, 12.12s/it][2022-12-20 14:20:27,062] [INFO] [timer.py:197:stop] 0/2330, RunningAvgSamplesPerSec=5.865950835960259, CurrSamplesPerSec=5.3302888245755655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1151/5000 [3:58:02<12:36:15, 11.79s/it][2022-12-20 14:20:38,107] [INFO] [timer.py:197:stop] 0/2332, RunningAvgSamplesPerSec=5.865945061077047, CurrSamplesPerSec=5.312614288253192, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1152/5000 [3:58:13<12:21:44, 11.57s/it][2022-12-20 14:20:49,050] [INFO] [timer.py:197:stop] 0/2334, RunningAvgSamplesPerSec=5.865987180419062, CurrSamplesPerSec=5.361222280309993, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1153/5000 [3:58:24<12:09:34, 11.38s/it][2022-12-20 14:21:00,040] [INFO] [timer.py:197:stop] 0/2336, RunningAvgSamplesPerSec=5.86600722984544, CurrSamplesPerSec=5.317433955159826, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1154/5000 [3:58:35<12:01:54, 11.26s/it][2022-12-20 14:21:11,047] [INFO] [timer.py:197:stop] 0/2338, RunningAvgSamplesPerSec=5.866019490246221, CurrSamplesPerSec=5.319441938212023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1155/5000 [3:58:46<11:56:49, 11.19s/it][2022-12-20 14:21:22,088] [INFO] [logging.py:68:log_dist] [Rank 0] step=1170, skipped=3, lr=[8.52e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:21:22,089] [INFO] [timer.py:197:stop] 0/2340, RunningAvgSamplesPerSec=5.866015352677619, CurrSamplesPerSec=5.314046709923048, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1156/5000 [3:58:57<11:53:52, 11.14s/it][2022-12-20 14:21:33,177] [INFO] [timer.py:197:stop] 0/2342, RunningAvgSamplesPerSec=5.865990808005341, CurrSamplesPerSec=5.281120053946874, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1157/5000 [3:59:08<11:52:37, 11.13s/it][2022-12-20 14:21:44,180] [INFO] [timer.py:197:stop] 0/2344, RunningAvgSamplesPerSec=5.866004904138269, CurrSamplesPerSec=5.350195730709113, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1158/5000 [3:59:19<11:50:04, 11.09s/it][2022-12-20 14:21:55,133] [INFO] [timer.py:197:stop] 0/2346, RunningAvgSamplesPerSec=5.866042003931394, CurrSamplesPerSec=5.366760127392136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1159/5000 [3:59:30<11:47:16, 11.05s/it][2022-12-20 14:22:06,032] [INFO] [timer.py:197:stop] 0/2348, RunningAvgSamplesPerSec=5.86610316435128, CurrSamplesPerSec=5.400620739877447, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1160/5000 [3:59:41<11:44:14, 11.00s/it][2022-12-20 14:22:17,047] [INFO] [timer.py:197:stop] 0/2350, RunningAvgSamplesPerSec=5.866111642478457, CurrSamplesPerSec=5.337300388167726, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1161/5000 [3:59:52<11:44:16, 11.01s/it][2022-12-20 14:22:28,038] [INFO] [timer.py:197:stop] 0/2352, RunningAvgSamplesPerSec=5.866130688903831, CurrSamplesPerSec=5.355272527018808, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1162/5000 [4:00:03<11:43:46, 11.00s/it][2022-12-20 14:22:39,079] [INFO] [timer.py:197:stop] 0/2354, RunningAvgSamplesPerSec=5.86612719306718, CurrSamplesPerSec=5.3206898937092735, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1163/5000 [4:00:14<11:44:19, 11.01s/it][2022-12-20 14:22:50,039] [INFO] [timer.py:197:stop] 0/2356, RunningAvgSamplesPerSec=5.866161114379853, CurrSamplesPerSec=5.359317024101602, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1164/5000 [4:00:25<11:43:07, 11.00s/it][2022-12-20 14:23:01,028] [INFO] [timer.py:197:stop] 0/2358, RunningAvgSamplesPerSec=5.866181281292867, CurrSamplesPerSec=5.348691748106546, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1165/5000 [4:00:36<11:42:45, 10.99s/it][2022-12-20 14:23:12,042] [INFO] [logging.py:68:log_dist] [Rank 0] step=1180, skipped=3, lr=[8.497777777777777e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:23:12,043] [INFO] [timer.py:197:stop] 0/2360, RunningAvgSamplesPerSec=5.8661896824405035, CurrSamplesPerSec=5.327635818212209, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1166/5000 [4:00:47<11:42:58, 11.00s/it][2022-12-20 14:23:23,018] [INFO] [timer.py:197:stop] 0/2362, RunningAvgSamplesPerSec=5.866217316938825, CurrSamplesPerSec=5.349279467119528, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1167/5000 [4:00:58<11:42:16, 10.99s/it][2022-12-20 14:23:34,023] [INFO] [timer.py:197:stop] 0/2364, RunningAvgSamplesPerSec=5.866230198120765, CurrSamplesPerSec=5.331378382058555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1168/5000 [4:01:09<11:42:19, 11.00s/it][2022-12-20 14:23:45,074] [INFO] [timer.py:197:stop] 0/2366, RunningAvgSamplesPerSec=5.866221819988924, CurrSamplesPerSec=5.306074308673665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1169/5000 [4:01:20<11:43:11, 11.01s/it][2022-12-20 14:23:56,091] [INFO] [timer.py:197:stop] 0/2368, RunningAvgSamplesPerSec=5.866228810395212, CurrSamplesPerSec=5.322969693102323, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1170/5000 [4:01:31<11:43:04, 11.01s/it][2022-12-20 14:24:07,126] [INFO] [timer.py:197:stop] 0/2370, RunningAvgSamplesPerSec=5.866234434708928, CurrSamplesPerSec=5.301017878224371, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1171/5000 [4:01:42<11:43:18, 11.02s/it][2022-12-20 14:24:18,150] [INFO] [timer.py:197:stop] 0/2372, RunningAvgSamplesPerSec=5.866248521951817, CurrSamplesPerSec=5.341322450572729, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1172/5000 [4:01:53<11:43:10, 11.02s/it][2022-12-20 14:24:29,202] [INFO] [timer.py:197:stop] 0/2374, RunningAvgSamplesPerSec=5.866240058534632, CurrSamplesPerSec=5.296656163808751, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1173/5000 [4:02:04<11:43:34, 11.03s/it][2022-12-20 14:24:40,175] [INFO] [timer.py:197:stop] 0/2376, RunningAvgSamplesPerSec=5.8662677316419645, CurrSamplesPerSec=5.33105926032227, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 23%|██▎       | 1174/5000 [4:02:15<11:42:16, 11.01s/it][2022-12-20 14:24:51,196] [INFO] [timer.py:197:stop] 0/2378, RunningAvgSamplesPerSec=5.866273388393853, CurrSamplesPerSec=5.320795779667754, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1175/5000 [4:02:26<11:42:15, 11.02s/it]
                                                        
{'loss': 0.0015, 'learning_rate': 8.477777777777778e-06, 'epoch': 28.65}

 24%|██▎       | 1175/5000 [4:02:26<11:42:15, 11.02s/it][2022-12-20 14:25:02,235] [INFO] [logging.py:68:log_dist] [Rank 0] step=1190, skipped=3, lr=[8.475555555555555e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:25:02,236] [INFO] [timer.py:197:stop] 0/2380, RunningAvgSamplesPerSec=5.866270932574624, CurrSamplesPerSec=5.328444410108352, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1176/5000 [4:02:37<11:42:31, 11.02s/it][2022-12-20 14:25:13,250] [INFO] [timer.py:197:stop] 0/2382, RunningAvgSamplesPerSec=5.866279706920595, CurrSamplesPerSec=5.3410852407781055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1177/5000 [4:02:48<11:42:09, 11.02s/it][2022-12-20 14:25:24,316] [INFO] [timer.py:197:stop] 0/2384, RunningAvgSamplesPerSec=5.866264919998554, CurrSamplesPerSec=5.294317813314271, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1178/5000 [4:02:59<11:42:52, 11.03s/it][2022-12-20 14:25:35,379] [INFO] [timer.py:197:stop] 0/2386, RunningAvgSamplesPerSec=5.866259420389441, CurrSamplesPerSec=5.277050551398603, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1179/5000 [4:03:10<11:43:14, 11.04s/it][2022-12-20 14:25:46,448] [INFO] [timer.py:197:stop] 0/2388, RunningAvgSamplesPerSec=5.866258168720059, CurrSamplesPerSec=5.319883653419607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1180/5000 [4:03:22<11:43:34, 11.05s/it][2022-12-20 14:25:57,559] [INFO] [timer.py:197:stop] 0/2390, RunningAvgSamplesPerSec=5.866238173802574, CurrSamplesPerSec=5.3043575034651615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1181/5000 [4:03:33<11:44:32, 11.07s/it][2022-12-20 14:26:08,599] [INFO] [timer.py:197:stop] 0/2392, RunningAvgSamplesPerSec=5.866243090994037, CurrSamplesPerSec=5.324683571295812, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1182/5000 [4:03:44<11:43:46, 11.06s/it][2022-12-20 14:26:19,559] [INFO] [timer.py:197:stop] 0/2394, RunningAvgSamplesPerSec=5.866277685655989, CurrSamplesPerSec=5.340624909849068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1183/5000 [4:03:55<11:41:42, 11.03s/it][2022-12-20 14:26:30,558] [INFO] [timer.py:197:stop] 0/2396, RunningAvgSamplesPerSec=5.866301360689517, CurrSamplesPerSec=5.354256909749205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1184/5000 [4:04:06<11:40:53, 11.02s/it][2022-12-20 14:26:41,562] [INFO] [timer.py:197:stop] 0/2398, RunningAvgSamplesPerSec=5.866314006200123, CurrSamplesPerSec=5.344340211278222, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1185/5000 [4:04:17<11:40:24, 11.02s/it][2022-12-20 14:26:52,577] [INFO] [logging.py:68:log_dist] [Rank 0] step=1200, skipped=3, lr=[8.453333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:26:52,578] [INFO] [timer.py:197:stop] 0/2400, RunningAvgSamplesPerSec=5.866321830021514, CurrSamplesPerSec=5.339912254788865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1186/5000 [4:04:28<11:40:14, 11.02s/it][2022-12-20 14:27:03,578] [INFO] [timer.py:197:stop] 0/2402, RunningAvgSamplesPerSec=5.866342924130218, CurrSamplesPerSec=5.363667915799298, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▎       | 1187/5000 [4:04:39<11:39:45, 11.01s/it][2022-12-20 14:27:14,670] [INFO] [timer.py:197:stop] 0/2404, RunningAvgSamplesPerSec=5.866323472040068, CurrSamplesPerSec=5.322167404705935, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1188/5000 [4:04:50<11:41:06, 11.04s/it][2022-12-20 14:27:25,680] [INFO] [timer.py:197:stop] 0/2406, RunningAvgSamplesPerSec=5.866333335281994, CurrSamplesPerSec=5.3121636873863425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1189/5000 [4:05:01<11:40:26, 11.03s/it][2022-12-20 14:27:35,805] [INFO] [timer.py:197:stop] 0/2408, RunningAvgSamplesPerSec=5.866735126668492, CurrSamplesPerSec=5.319866362948497, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1190/5000 [4:05:16<12:57:50, 12.25s/it][2022-12-20 14:27:46,938] [INFO] [timer.py:197:stop] 0/2410, RunningAvgSamplesPerSec=5.866690003407645, CurrSamplesPerSec=5.250663287908185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1191/5000 [4:05:27<12:35:58, 11.91s/it][2022-12-20 14:27:57,975] [INFO] [timer.py:197:stop] 0/2412, RunningAvgSamplesPerSec=5.86668728388693, CurrSamplesPerSec=5.315811269425496, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1192/5000 [4:05:38<12:18:28, 11.64s/it][2022-12-20 14:28:08,984] [INFO] [timer.py:197:stop] 0/2414, RunningAvgSamplesPerSec=5.866697793970734, CurrSamplesPerSec=5.308342857359818, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1193/5000 [4:05:49<12:07:30, 11.47s/it][2022-12-20 14:28:20,114] [INFO] [timer.py:197:stop] 0/2416, RunningAvgSamplesPerSec=5.866654404791631, CurrSamplesPerSec=5.256519641806495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1194/5000 [4:06:00<11:59:38, 11.34s/it][2022-12-20 14:28:31,105] [INFO] [timer.py:197:stop] 0/2418, RunningAvgSamplesPerSec=5.866673234860263, CurrSamplesPerSec=5.319040557558386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1195/5000 [4:06:11<11:52:38, 11.24s/it][2022-12-20 14:28:42,060] [INFO] [logging.py:68:log_dist] [Rank 0] step=1210, skipped=3, lr=[8.431111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:28:42,062] [INFO] [timer.py:197:stop] 0/2420, RunningAvgSamplesPerSec=5.866706810744663, CurrSamplesPerSec=5.345323543310957, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1196/5000 [4:06:22<11:47:59, 11.17s/it][2022-12-20 14:28:53,431] [INFO] [timer.py:197:stop] 0/2422, RunningAvgSamplesPerSec=5.866557098315779, CurrSamplesPerSec=5.036977225099463, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1197/5000 [4:06:34<11:52:04, 11.23s/it][2022-12-20 14:29:04,476] [INFO] [timer.py:197:stop] 0/2424, RunningAvgSamplesPerSec=5.866551769893157, CurrSamplesPerSec=5.3283517575204, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1198/5000 [4:06:45<11:47:47, 11.17s/it][2022-12-20 14:29:15,513] [INFO] [timer.py:197:stop] 0/2426, RunningAvgSamplesPerSec=5.866549631747129, CurrSamplesPerSec=5.311750791324563, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1199/5000 [4:06:56<11:45:03, 11.13s/it][2022-12-20 14:29:26,854] [INFO] [timer.py:197:stop] 0/2428, RunningAvgSamplesPerSec=5.866413203666896, CurrSamplesPerSec=5.056082632148807, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1200/5000 [4:07:07<11:49:20, 11.20s/it]
                                                        
{'loss': 0.0012, 'learning_rate': 8.422222222222223e-06, 'epoch': 29.27}

 24%|██▍       | 1200/5000 [4:07:07<11:49:20, 11.20s/it][2022-12-20 14:29:37,931] [INFO] [timer.py:197:stop] 0/2430, RunningAvgSamplesPerSec=5.866393851256666, CurrSamplesPerSec=5.296994802408528, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1201/5000 [4:07:18<11:46:32, 11.16s/it][2022-12-20 14:29:48,985] [INFO] [timer.py:197:stop] 0/2432, RunningAvgSamplesPerSec=5.86638361714413, CurrSamplesPerSec=5.302136763312669, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1202/5000 [4:07:29<11:43:39, 11.12s/it][2022-12-20 14:30:00,129] [INFO] [timer.py:197:stop] 0/2434, RunningAvgSamplesPerSec=5.866334355478099, CurrSamplesPerSec=5.194048236660044, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1203/5000 [4:07:40<11:44:08, 11.13s/it][2022-12-20 14:30:11,139] [INFO] [timer.py:197:stop] 0/2436, RunningAvgSamplesPerSec=5.866343779590564, CurrSamplesPerSec=5.315572109716659, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1204/5000 [4:07:51<11:42:06, 11.10s/it][2022-12-20 14:30:22,192] [INFO] [timer.py:197:stop] 0/2438, RunningAvgSamplesPerSec=5.866340029215908, CurrSamplesPerSec=5.295177531307941, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1205/5000 [4:08:02<11:41:15, 11.09s/it][2022-12-20 14:30:33,268] [INFO] [logging.py:68:log_dist] [Rank 0] step=1220, skipped=3, lr=[8.40888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:30:33,270] [INFO] [timer.py:197:stop] 0/2440, RunningAvgSamplesPerSec=5.866319653101191, CurrSamplesPerSec=5.281954496344795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1206/5000 [4:08:13<11:40:44, 11.08s/it][2022-12-20 14:30:44,300] [INFO] [timer.py:197:stop] 0/2442, RunningAvgSamplesPerSec=5.866320287619543, CurrSamplesPerSec=5.3164258991089115, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1207/5000 [4:08:24<11:39:48, 11.07s/it][2022-12-20 14:30:55,313] [INFO] [timer.py:197:stop] 0/2444, RunningAvgSamplesPerSec=5.8663287960797925, CurrSamplesPerSec=5.342777840893666, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1208/5000 [4:08:35<11:37:38, 11.04s/it][2022-12-20 14:31:06,448] [INFO] [timer.py:197:stop] 0/2446, RunningAvgSamplesPerSec=5.866283189551714, CurrSamplesPerSec=5.194655537071338, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1209/5000 [4:08:46<11:40:02, 11.08s/it][2022-12-20 14:31:17,498] [INFO] [timer.py:197:stop] 0/2448, RunningAvgSamplesPerSec=5.8662757146520965, CurrSamplesPerSec=5.303761379649366, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1210/5000 [4:08:58<11:39:26, 11.07s/it][2022-12-20 14:31:28,490] [INFO] [timer.py:197:stop] 0/2450, RunningAvgSamplesPerSec=5.866293484110395, CurrSamplesPerSec=5.3618690908382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1211/5000 [4:09:09<11:37:08, 11.04s/it][2022-12-20 14:31:39,616] [INFO] [timer.py:197:stop] 0/2452, RunningAvgSamplesPerSec=5.866252045631158, CurrSamplesPerSec=5.2163481454814455, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1212/5000 [4:09:20<11:38:55, 11.07s/it][2022-12-20 14:31:50,605] [INFO] [timer.py:197:stop] 0/2454, RunningAvgSamplesPerSec=5.866271718732502, CurrSamplesPerSec=5.353360606871251, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1213/5000 [4:09:31<11:37:24, 11.05s/it][2022-12-20 14:32:01,654] [INFO] [timer.py:197:stop] 0/2456, RunningAvgSamplesPerSec=5.8662642275321, CurrSamplesPerSec=5.308947151348811, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1214/5000 [4:09:42<11:36:29, 11.04s/it][2022-12-20 14:32:12,653] [INFO] [timer.py:197:stop] 0/2458, RunningAvgSamplesPerSec=5.866279437545395, CurrSamplesPerSec=5.32057051340014, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1215/5000 [4:09:53<11:35:22, 11.02s/it][2022-12-20 14:32:23,644] [INFO] [logging.py:68:log_dist] [Rank 0] step=1230, skipped=3, lr=[8.386666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:32:23,645] [INFO] [timer.py:197:stop] 0/2460, RunningAvgSamplesPerSec=5.866296952599812, CurrSamplesPerSec=5.316530141259214, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1216/5000 [4:10:04<11:34:38, 11.01s/it][2022-12-20 14:32:34,607] [INFO] [timer.py:197:stop] 0/2462, RunningAvgSamplesPerSec=5.866327738215617, CurrSamplesPerSec=5.344379154561608, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1217/5000 [4:10:15<11:33:44, 11.00s/it][2022-12-20 14:32:45,643] [INFO] [timer.py:197:stop] 0/2464, RunningAvgSamplesPerSec=5.866326308963281, CurrSamplesPerSec=5.291882626406358, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1218/5000 [4:10:26<11:34:53, 11.02s/it][2022-12-20 14:32:56,692] [INFO] [timer.py:197:stop] 0/2466, RunningAvgSamplesPerSec=5.8663199379621895, CurrSamplesPerSec=5.314945680668286, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1219/5000 [4:10:37<11:34:01, 11.01s/it][2022-12-20 14:33:07,685] [INFO] [timer.py:197:stop] 0/2468, RunningAvgSamplesPerSec=5.8663375598948795, CurrSamplesPerSec=5.310083462141238, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1220/5000 [4:10:48<11:34:14, 11.02s/it][2022-12-20 14:33:18,828] [INFO] [timer.py:197:stop] 0/2470, RunningAvgSamplesPerSec=5.8662889443349036, CurrSamplesPerSec=5.215359809380851, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1221/5000 [4:10:59<11:36:04, 11.05s/it][2022-12-20 14:33:29,812] [INFO] [timer.py:197:stop] 0/2472, RunningAvgSamplesPerSec=5.866310152878439, CurrSamplesPerSec=5.3391466918986925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1222/5000 [4:11:10<11:34:11, 11.02s/it][2022-12-20 14:33:40,755] [INFO] [timer.py:197:stop] 0/2474, RunningAvgSamplesPerSec=5.866349132494918, CurrSamplesPerSec=5.3568706501380206, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1223/5000 [4:11:21<11:33:22, 11.01s/it][2022-12-20 14:33:51,773] [INFO] [timer.py:197:stop] 0/2476, RunningAvgSamplesPerSec=5.866355595508482, CurrSamplesPerSec=5.333332273695414, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1224/5000 [4:11:32<11:32:46, 11.01s/it][2022-12-20 14:34:02,780] [INFO] [timer.py:197:stop] 0/2478, RunningAvgSamplesPerSec=5.866366271959182, CurrSamplesPerSec=5.319785394432974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 24%|██▍       | 1225/5000 [4:11:43<11:33:01, 11.01s/it]
                                                        
{'loss': 0.0011, 'learning_rate': 8.366666666666667e-06, 'epoch': 29.87}

 24%|██▍       | 1225/5000 [4:11:43<11:33:01, 11.01s/it][2022-12-20 14:34:13,832] [INFO] [logging.py:68:log_dist] [Rank 0] step=1240, skipped=3, lr=[8.364444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:34:13,833] [INFO] [timer.py:197:stop] 0/2480, RunningAvgSamplesPerSec=5.8663624963996766, CurrSamplesPerSec=5.3107463596516356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1226/5000 [4:11:54<11:33:38, 11.03s/it][2022-12-20 14:34:24,879] [INFO] [timer.py:197:stop] 0/2482, RunningAvgSamplesPerSec=5.866356473048373, CurrSamplesPerSec=5.309833684110582, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1227/5000 [4:12:05<11:34:38, 11.05s/it][2022-12-20 14:34:35,965] [INFO] [timer.py:197:stop] 0/2484, RunningAvgSamplesPerSec=5.866338980521272, CurrSamplesPerSec=5.313235540486574, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1228/5000 [4:12:16<11:34:07, 11.04s/it][2022-12-20 14:34:47,008] [INFO] [timer.py:197:stop] 0/2486, RunningAvgSamplesPerSec=5.866342291838639, CurrSamplesPerSec=5.31729133814346, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1229/5000 [4:12:27<11:34:16, 11.05s/it][2022-12-20 14:34:58,150] [INFO] [timer.py:197:stop] 0/2488, RunningAvgSamplesPerSec=5.86629436505756, CurrSamplesPerSec=5.228989271702679, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1230/5000 [4:12:38<11:36:06, 11.08s/it][2022-12-20 14:35:08,289] [INFO] [timer.py:197:stop] 0/2490, RunningAvgSamplesPerSec=5.866683051449626, CurrSamplesPerSec=6.255712617522649, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 14:35:19,365] [INFO] [timer.py:197:stop] 0/2492, RunningAvgSamplesPerSec=5.866662220221625, CurrSamplesPerSec=5.290438771318746, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1231/5000 [4:12:54<13:13:15, 12.63s/it][2022-12-20 14:35:30,406] [INFO] [timer.py:197:stop] 0/2494, RunningAvgSamplesPerSec=5.866658069700753, CurrSamplesPerSec=5.317211711881955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1232/5000 [4:13:06<12:43:09, 12.15s/it][2022-12-20 14:35:41,401] [INFO] [timer.py:197:stop] 0/2496, RunningAvgSamplesPerSec=5.866674264185871, CurrSamplesPerSec=5.3262005007554505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1233/5000 [4:13:16<12:21:09, 11.80s/it][2022-12-20 14:35:52,400] [INFO] [timer.py:197:stop] 0/2498, RunningAvgSamplesPerSec=5.866688874988825, CurrSamplesPerSec=5.356402891484871, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1234/5000 [4:13:27<12:05:46, 11.56s/it][2022-12-20 14:36:03,580] [INFO] [logging.py:68:log_dist] [Rank 0] step=1250, skipped=3, lr=[8.342222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:36:03,582] [INFO] [timer.py:197:stop] 0/2500, RunningAvgSamplesPerSec=5.866624530729723, CurrSamplesPerSec=5.182571068299581, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1235/5000 [4:13:39<11:58:24, 11.45s/it][2022-12-20 14:36:14,516] [INFO] [timer.py:197:stop] 0/2502, RunningAvgSamplesPerSec=5.866667110443997, CurrSamplesPerSec=5.365425908578023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1236/5000 [4:13:50<11:48:32, 11.29s/it][2022-12-20 14:36:25,557] [INFO] [timer.py:197:stop] 0/2504, RunningAvgSamplesPerSec=5.866662660511366, CurrSamplesPerSec=5.325509439508219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1237/5000 [4:14:01<11:43:35, 11.22s/it][2022-12-20 14:36:36,824] [INFO] [timer.py:197:stop] 0/2506, RunningAvgSamplesPerSec=5.866565210964884, CurrSamplesPerSec=5.142926618483068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1238/5000 [4:14:12<11:44:27, 11.24s/it][2022-12-20 14:36:47,913] [INFO] [timer.py:197:stop] 0/2508, RunningAvgSamplesPerSec=5.866572477495271, CurrSamplesPerSec=5.318034210175345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1239/5000 [4:14:23<11:41:22, 11.19s/it][2022-12-20 14:36:58,900] [INFO] [timer.py:197:stop] 0/2510, RunningAvgSamplesPerSec=5.866600153879609, CurrSamplesPerSec=5.338725555983464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1240/5000 [4:14:34<11:37:22, 11.13s/it][2022-12-20 14:37:10,314] [INFO] [timer.py:197:stop] 0/2512, RunningAvgSamplesPerSec=5.86643679765559, CurrSamplesPerSec=4.98516155769956, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1241/5000 [4:14:45<11:42:33, 11.21s/it][2022-12-20 14:37:21,316] [INFO] [timer.py:197:stop] 0/2514, RunningAvgSamplesPerSec=5.866449943668701, CurrSamplesPerSec=5.3272116329982016, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1242/5000 [4:14:56<11:38:23, 11.15s/it][2022-12-20 14:37:32,338] [INFO] [timer.py:197:stop] 0/2516, RunningAvgSamplesPerSec=5.866461011796087, CurrSamplesPerSec=5.310245021590694, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1243/5000 [4:15:07<11:35:47, 11.11s/it][2022-12-20 14:37:43,454] [INFO] [timer.py:197:stop] 0/2518, RunningAvgSamplesPerSec=5.866425892772194, CurrSamplesPerSec=5.212907613883006, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1244/5000 [4:15:19<11:35:43, 11.11s/it][2022-12-20 14:37:54,473] [INFO] [logging.py:68:log_dist] [Rank 0] step=1260, skipped=3, lr=[8.32e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:37:54,474] [INFO] [timer.py:197:stop] 0/2520, RunningAvgSamplesPerSec=5.86644255249078, CurrSamplesPerSec=5.332693176822377, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1245/5000 [4:15:30<11:33:44, 11.09s/it][2022-12-20 14:38:05,517] [INFO] [timer.py:197:stop] 0/2522, RunningAvgSamplesPerSec=5.866444891204957, CurrSamplesPerSec=5.3370699022057435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1246/5000 [4:15:41<11:32:47, 11.07s/it][2022-12-20 14:38:16,534] [INFO] [timer.py:197:stop] 0/2524, RunningAvgSamplesPerSec=5.866458659194212, CurrSamplesPerSec=5.328032998001378, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1247/5000 [4:15:52<11:31:31, 11.06s/it][2022-12-20 14:38:27,613] [INFO] [timer.py:197:stop] 0/2526, RunningAvgSamplesPerSec=5.866439026504298, CurrSamplesPerSec=5.311446626378874, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1248/5000 [4:16:03<11:31:46, 11.06s/it][2022-12-20 14:38:38,668] [INFO] [timer.py:197:stop] 0/2528, RunningAvgSamplesPerSec=5.866428974068871, CurrSamplesPerSec=5.300253170155539, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▍       | 1249/5000 [4:16:14<11:31:27, 11.06s/it][2022-12-20 14:38:49,702] [INFO] [timer.py:197:stop] 0/2530, RunningAvgSamplesPerSec=5.866428517493221, CurrSamplesPerSec=5.314769103190194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1250/5000 [4:16:25<11:30:46, 11.05s/it]
                                                        
{'loss': 0.0009, 'learning_rate': 8.30888888888889e-06, 'epoch': 30.48}

 25%|██▌       | 1250/5000 [4:16:25<11:30:46, 11.05s/it][2022-12-20 14:39:01,184] [INFO] [timer.py:197:stop] 0/2532, RunningAvgSamplesPerSec=5.866420303426205, CurrSamplesPerSec=5.316974111182338, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1251/5000 [4:16:36<11:38:38, 11.18s/it][2022-12-20 14:39:12,230] [INFO] [timer.py:197:stop] 0/2534, RunningAvgSamplesPerSec=5.866414357189164, CurrSamplesPerSec=5.304399430014414, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1252/5000 [4:16:47<11:35:55, 11.14s/it][2022-12-20 14:39:23,285] [INFO] [timer.py:197:stop] 0/2536, RunningAvgSamplesPerSec=5.866405074679397, CurrSamplesPerSec=5.309512095365863, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1253/5000 [4:16:58<11:34:07, 11.11s/it][2022-12-20 14:39:34,592] [INFO] [timer.py:197:stop] 0/2538, RunningAvgSamplesPerSec=5.866398287009394, CurrSamplesPerSec=5.310256366836915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1254/5000 [4:17:10<11:37:32, 11.17s/it][2022-12-20 14:39:45,600] [INFO] [logging.py:68:log_dist] [Rank 0] step=1270, skipped=3, lr=[8.297777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:39:45,602] [INFO] [timer.py:197:stop] 0/2540, RunningAvgSamplesPerSec=5.866407689182847, CurrSamplesPerSec=5.316591003717835, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1255/5000 [4:17:21<11:34:18, 11.12s/it][2022-12-20 14:39:56,650] [INFO] [timer.py:197:stop] 0/2542, RunningAvgSamplesPerSec=5.866400554596771, CurrSamplesPerSec=5.317751658378555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1256/5000 [4:17:32<11:32:42, 11.10s/it][2022-12-20 14:40:07,743] [INFO] [timer.py:197:stop] 0/2544, RunningAvgSamplesPerSec=5.866374632998121, CurrSamplesPerSec=5.325266025782945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1257/5000 [4:17:43<11:32:22, 11.10s/it][2022-12-20 14:40:18,749] [INFO] [timer.py:197:stop] 0/2546, RunningAvgSamplesPerSec=5.866385946310674, CurrSamplesPerSec=5.3002286813726345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1258/5000 [4:17:54<11:30:26, 11.07s/it][2022-12-20 14:40:29,709] [INFO] [timer.py:197:stop] 0/2548, RunningAvgSamplesPerSec=5.866416185594421, CurrSamplesPerSec=5.34435063865912, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1259/5000 [4:18:05<11:28:11, 11.04s/it][2022-12-20 14:40:40,656] [INFO] [timer.py:197:stop] 0/2550, RunningAvgSamplesPerSec=5.866452290321615, CurrSamplesPerSec=5.360355970084697, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1260/5000 [4:18:16<11:26:18, 11.01s/it][2022-12-20 14:40:51,640] [INFO] [timer.py:197:stop] 0/2552, RunningAvgSamplesPerSec=5.86647274191724, CurrSamplesPerSec=5.34167554108762, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1261/5000 [4:18:27<11:25:38, 11.00s/it][2022-12-20 14:41:02,695] [INFO] [timer.py:197:stop] 0/2554, RunningAvgSamplesPerSec=5.866463119436804, CurrSamplesPerSec=5.319463442456576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1262/5000 [4:18:38<11:26:27, 11.02s/it][2022-12-20 14:41:13,733] [INFO] [timer.py:197:stop] 0/2556, RunningAvgSamplesPerSec=5.866464812760032, CurrSamplesPerSec=5.3356748896233634, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1263/5000 [4:18:49<11:26:37, 11.02s/it][2022-12-20 14:41:24,798] [INFO] [timer.py:197:stop] 0/2558, RunningAvgSamplesPerSec=5.86645108537351, CurrSamplesPerSec=5.304089608463672, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1264/5000 [4:19:00<11:27:12, 11.04s/it][2022-12-20 14:41:35,884] [INFO] [logging.py:68:log_dist] [Rank 0] step=1280, skipped=3, lr=[8.275555555555557e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:41:35,885] [INFO] [timer.py:197:stop] 0/2560, RunningAvgSamplesPerSec=5.866435162823652, CurrSamplesPerSec=5.304401735993852, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1265/5000 [4:19:11<11:27:57, 11.05s/it][2022-12-20 14:41:46,916] [INFO] [timer.py:197:stop] 0/2562, RunningAvgSamplesPerSec=5.866438772156116, CurrSamplesPerSec=5.352324370543916, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1266/5000 [4:19:22<11:27:22, 11.05s/it][2022-12-20 14:41:57,907] [INFO] [timer.py:197:stop] 0/2564, RunningAvgSamplesPerSec=5.866456043934021, CurrSamplesPerSec=5.3387862905984536, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1267/5000 [4:19:33<11:26:11, 11.03s/it][2022-12-20 14:42:08,976] [INFO] [timer.py:197:stop] 0/2566, RunningAvgSamplesPerSec=5.866445757930909, CurrSamplesPerSec=5.311216055988494, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1268/5000 [4:19:44<11:26:45, 11.04s/it][2022-12-20 14:42:20,010] [INFO] [timer.py:197:stop] 0/2568, RunningAvgSamplesPerSec=5.86644502153176, CurrSamplesPerSec=5.296935223960517, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1269/5000 [4:19:55<11:26:26, 11.04s/it][2022-12-20 14:42:30,995] [INFO] [timer.py:197:stop] 0/2570, RunningAvgSamplesPerSec=5.8664646152671525, CurrSamplesPerSec=5.337238201688508, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1270/5000 [4:20:06<11:25:15, 11.02s/it][2022-12-20 14:42:42,041] [INFO] [timer.py:197:stop] 0/2572, RunningAvgSamplesPerSec=5.866465979708828, CurrSamplesPerSec=5.329003355478731, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1271/5000 [4:20:17<11:25:29, 11.03s/it][2022-12-20 14:42:52,214] [INFO] [timer.py:197:stop] 0/2574, RunningAvgSamplesPerSec=5.866821180582034, CurrSamplesPerSec=5.272568266615608, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1272/5000 [4:20:32<12:40:56, 12.25s/it][2022-12-20 14:43:03,210] [INFO] [timer.py:197:stop] 0/2576, RunningAvgSamplesPerSec=5.866835855180823, CurrSamplesPerSec=5.317548138551296, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1273/5000 [4:20:43<12:18:13, 11.88s/it][2022-12-20 14:43:14,291] [INFO] [timer.py:197:stop] 0/2578, RunningAvgSamplesPerSec=5.866815882840631, CurrSamplesPerSec=5.282412252696115, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 25%|██▌       | 1274/5000 [4:20:54<12:03:05, 11.64s/it][2022-12-20 14:43:25,457] [INFO] [logging.py:68:log_dist] [Rank 0] step=1290, skipped=3, lr=[8.253333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:43:25,458] [INFO] [timer.py:197:stop] 0/2580, RunningAvgSamplesPerSec=5.866814189672369, CurrSamplesPerSec=5.321991613220825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1275/5000 [4:21:06<11:54:21, 11.51s/it]
                                                        
{'loss': 0.0011, 'learning_rate': 8.253333333333334e-06, 'epoch': 31.1}

 26%|██▌       | 1275/5000 [4:21:06<11:54:21, 11.51s/it][2022-12-20 14:43:36,493] [INFO] [timer.py:197:stop] 0/2582, RunningAvgSamplesPerSec=5.866813256736886, CurrSamplesPerSec=5.33998958799783, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1276/5000 [4:21:17<11:44:33, 11.35s/it][2022-12-20 14:43:47,534] [INFO] [timer.py:197:stop] 0/2584, RunningAvgSamplesPerSec=5.866809048074333, CurrSamplesPerSec=5.293974714236095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1277/5000 [4:21:28<11:39:37, 11.28s/it][2022-12-20 14:43:58,615] [INFO] [timer.py:197:stop] 0/2586, RunningAvgSamplesPerSec=5.866788005198868, CurrSamplesPerSec=5.307037103230776, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1278/5000 [4:21:39<11:35:32, 11.21s/it][2022-12-20 14:44:09,685] [INFO] [timer.py:197:stop] 0/2588, RunningAvgSamplesPerSec=5.86677112614875, CurrSamplesPerSec=5.303040927825822, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1279/5000 [4:21:50<11:32:35, 11.17s/it][2022-12-20 14:44:20,691] [INFO] [timer.py:197:stop] 0/2590, RunningAvgSamplesPerSec=5.866781356573359, CurrSamplesPerSec=5.354594836378892, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1280/5000 [4:22:01<11:29:14, 11.12s/it][2022-12-20 14:44:31,852] [INFO] [timer.py:197:stop] 0/2592, RunningAvgSamplesPerSec=5.866727194956834, CurrSamplesPerSec=5.213148153521151, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1281/5000 [4:22:12<11:29:47, 11.13s/it][2022-12-20 14:44:42,871] [INFO] [timer.py:197:stop] 0/2594, RunningAvgSamplesPerSec=5.866731793977151, CurrSamplesPerSec=5.3322245466290905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1282/5000 [4:22:23<11:27:39, 11.10s/it][2022-12-20 14:44:53,909] [INFO] [timer.py:197:stop] 0/2596, RunningAvgSamplesPerSec=5.866732904322087, CurrSamplesPerSec=5.319831571487487, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1283/5000 [4:22:34<11:26:25, 11.08s/it][2022-12-20 14:45:05,036] [INFO] [timer.py:197:stop] 0/2598, RunningAvgSamplesPerSec=5.866692714824776, CurrSamplesPerSec=5.244622402872517, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1284/5000 [4:22:45<11:26:30, 11.08s/it][2022-12-20 14:45:16,066] [INFO] [logging.py:68:log_dist] [Rank 0] step=1300, skipped=3, lr=[8.231111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:45:16,068] [INFO] [timer.py:197:stop] 0/2600, RunningAvgSamplesPerSec=5.8666919755742715, CurrSamplesPerSec=5.298228900231777, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1285/5000 [4:22:56<11:26:13, 11.08s/it][2022-12-20 14:45:27,151] [INFO] [timer.py:197:stop] 0/2602, RunningAvgSamplesPerSec=5.8666760184045845, CurrSamplesPerSec=5.296026452959583, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1286/5000 [4:23:07<11:24:50, 11.06s/it][2022-12-20 14:45:38,236] [INFO] [timer.py:197:stop] 0/2604, RunningAvgSamplesPerSec=5.866653736164829, CurrSamplesPerSec=5.238799875315857, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1287/5000 [4:23:18<11:25:17, 11.07s/it][2022-12-20 14:45:49,220] [INFO] [timer.py:197:stop] 0/2606, RunningAvgSamplesPerSec=5.866673237070117, CurrSamplesPerSec=5.338110855054394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1288/5000 [4:23:29<11:23:54, 11.05s/it][2022-12-20 14:46:00,247] [INFO] [timer.py:197:stop] 0/2608, RunningAvgSamplesPerSec=5.8666750357810615, CurrSamplesPerSec=5.32300663669837, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1289/5000 [4:23:40<11:23:56, 11.06s/it][2022-12-20 14:46:11,589] [INFO] [timer.py:197:stop] 0/2610, RunningAvgSamplesPerSec=5.86667991060729, CurrSamplesPerSec=5.361265538925755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1290/5000 [4:23:52<11:28:27, 11.13s/it][2022-12-20 14:46:22,556] [INFO] [timer.py:197:stop] 0/2612, RunningAvgSamplesPerSec=5.866706986025993, CurrSamplesPerSec=5.384632123344585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1291/5000 [4:24:03<11:25:10, 11.08s/it][2022-12-20 14:46:33,567] [INFO] [timer.py:197:stop] 0/2614, RunningAvgSamplesPerSec=5.866715708805527, CurrSamplesPerSec=5.344920801683196, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1292/5000 [4:24:14<11:23:37, 11.06s/it][2022-12-20 14:46:44,580] [INFO] [timer.py:197:stop] 0/2616, RunningAvgSamplesPerSec=5.866723491105463, CurrSamplesPerSec=5.342253000998336, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1293/5000 [4:24:25<11:22:49, 11.05s/it][2022-12-20 14:46:55,613] [INFO] [timer.py:197:stop] 0/2618, RunningAvgSamplesPerSec=5.8667232221646355, CurrSamplesPerSec=5.339256287327342, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1294/5000 [4:24:36<11:20:58, 11.02s/it][2022-12-20 14:47:06,593] [INFO] [logging.py:68:log_dist] [Rank 0] step=1310, skipped=3, lr=[8.20888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:47:06,594] [INFO] [timer.py:197:stop] 0/2620, RunningAvgSamplesPerSec=5.866743855801862, CurrSamplesPerSec=5.321492580630581, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1295/5000 [4:24:47<11:20:08, 11.01s/it][2022-12-20 14:47:17,630] [INFO] [timer.py:197:stop] 0/2622, RunningAvgSamplesPerSec=5.866743818081918, CurrSamplesPerSec=5.281109456249781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1296/5000 [4:24:58<11:20:04, 11.02s/it][2022-12-20 14:47:28,609] [INFO] [timer.py:197:stop] 0/2624, RunningAvgSamplesPerSec=5.866765203046419, CurrSamplesPerSec=5.317507899933124, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1297/5000 [4:25:09<11:20:18, 11.02s/it][2022-12-20 14:47:39,585] [INFO] [timer.py:197:stop] 0/2626, RunningAvgSamplesPerSec=5.8667884842650135, CurrSamplesPerSec=5.374452474892769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1298/5000 [4:25:20<11:19:04, 11.01s/it][2022-12-20 14:47:50,648] [INFO] [timer.py:197:stop] 0/2628, RunningAvgSamplesPerSec=5.866776093047751, CurrSamplesPerSec=5.2886072401084485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1299/5000 [4:25:31<11:20:20, 11.03s/it][2022-12-20 14:48:01,698] [INFO] [timer.py:197:stop] 0/2630, RunningAvgSamplesPerSec=5.866769019160517, CurrSamplesPerSec=5.318527536248883, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1300/5000 [4:25:42<11:20:00, 11.03s/it]
                                                        
{'loss': 0.0008, 'learning_rate': 8.197777777777779e-06, 'epoch': 31.7}

 26%|██▌       | 1300/5000 [4:25:42<11:20:00, 11.03s/it][2022-12-20 14:48:12,719] [INFO] [timer.py:197:stop] 0/2632, RunningAvgSamplesPerSec=5.866774325392706, CurrSamplesPerSec=5.320704869355892, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1301/5000 [4:25:53<11:19:45, 11.03s/it][2022-12-20 14:48:23,742] [INFO] [timer.py:197:stop] 0/2634, RunningAvgSamplesPerSec=5.8667787109203395, CurrSamplesPerSec=5.3207086660120915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1302/5000 [4:26:04<11:19:50, 11.03s/it][2022-12-20 14:48:34,803] [INFO] [timer.py:197:stop] 0/2636, RunningAvgSamplesPerSec=5.8667663860237615, CurrSamplesPerSec=5.300755135405741, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1303/5000 [4:26:15<11:19:04, 11.02s/it][2022-12-20 14:48:45,758] [INFO] [timer.py:197:stop] 0/2638, RunningAvgSamplesPerSec=5.8667983476471495, CurrSamplesPerSec=5.34139684866578, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1304/5000 [4:26:26<11:18:31, 11.02s/it][2022-12-20 14:48:56,789] [INFO] [logging.py:68:log_dist] [Rank 0] step=1320, skipped=3, lr=[8.186666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:48:56,791] [INFO] [timer.py:197:stop] 0/2640, RunningAvgSamplesPerSec=5.866797300352509, CurrSamplesPerSec=5.3116742738501355, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1305/5000 [4:26:37<11:17:58, 11.01s/it][2022-12-20 14:49:07,755] [INFO] [timer.py:197:stop] 0/2642, RunningAvgSamplesPerSec=5.8668251450220525, CurrSamplesPerSec=5.34044662200489, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1306/5000 [4:26:48<11:17:35, 11.01s/it][2022-12-20 14:49:18,769] [INFO] [timer.py:197:stop] 0/2644, RunningAvgSamplesPerSec=5.866831687384138, CurrSamplesPerSec=5.324439810616205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1307/5000 [4:26:59<11:17:50, 11.01s/it][2022-12-20 14:49:29,898] [INFO] [timer.py:197:stop] 0/2646, RunningAvgSamplesPerSec=5.8667925884831496, CurrSamplesPerSec=5.239993500455335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1308/5000 [4:27:10<11:20:52, 11.07s/it][2022-12-20 14:49:40,971] [INFO] [timer.py:197:stop] 0/2648, RunningAvgSamplesPerSec=5.866824998765912, CurrSamplesPerSec=5.338632333161435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1309/5000 [4:27:21<11:18:44, 11.03s/it][2022-12-20 14:49:51,932] [INFO] [timer.py:197:stop] 0/2650, RunningAvgSamplesPerSec=5.866852764414854, CurrSamplesPerSec=5.33621053062638, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1310/5000 [4:27:32<11:17:20, 11.01s/it][2022-12-20 14:50:02,886] [INFO] [timer.py:197:stop] 0/2652, RunningAvgSamplesPerSec=5.866884156249551, CurrSamplesPerSec=5.348771041067318, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1311/5000 [4:27:43<11:17:38, 11.02s/it][2022-12-20 14:50:13,923] [INFO] [timer.py:197:stop] 0/2654, RunningAvgSamplesPerSec=5.866881730060096, CurrSamplesPerSec=5.35244582042356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▌       | 1312/5000 [4:27:54<11:16:08, 11.00s/it][2022-12-20 14:50:23,994] [INFO] [timer.py:197:stop] 0/2656, RunningAvgSamplesPerSec=5.867266908406734, CurrSamplesPerSec=6.262873907064042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 14:50:34,977] [INFO] [timer.py:197:stop] 0/2658, RunningAvgSamplesPerSec=5.867285090561456, CurrSamplesPerSec=5.340587083781192, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1313/5000 [4:28:10<12:50:49, 12.54s/it][2022-12-20 14:50:46,028] [INFO] [logging.py:68:log_dist] [Rank 0] step=1330, skipped=3, lr=[8.164444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:50:46,030] [INFO] [timer.py:197:stop] 0/2660, RunningAvgSamplesPerSec=5.867276005265758, CurrSamplesPerSec=5.328606031010614, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1314/5000 [4:28:21<12:23:08, 12.10s/it][2022-12-20 14:50:57,089] [INFO] [timer.py:197:stop] 0/2662, RunningAvgSamplesPerSec=5.867264634360738, CurrSamplesPerSec=5.288355727730571, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1315/5000 [4:28:32<12:03:48, 11.79s/it][2022-12-20 14:51:08,077] [INFO] [timer.py:197:stop] 0/2664, RunningAvgSamplesPerSec=5.8672822486468466, CurrSamplesPerSec=5.299272749167656, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1316/5000 [4:28:43<11:48:56, 11.55s/it][2022-12-20 14:51:19,214] [INFO] [timer.py:197:stop] 0/2666, RunningAvgSamplesPerSec=5.867239660302382, CurrSamplesPerSec=5.3288338823649735, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1317/5000 [4:28:54<11:41:12, 11.42s/it][2022-12-20 14:51:30,209] [INFO] [timer.py:197:stop] 0/2668, RunningAvgSamplesPerSec=5.867254224246213, CurrSamplesPerSec=5.3139932694613945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1318/5000 [4:29:05<11:33:08, 11.29s/it][2022-12-20 14:51:41,232] [INFO] [timer.py:197:stop] 0/2670, RunningAvgSamplesPerSec=5.867257901401531, CurrSamplesPerSec=5.331558818155095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1319/5000 [4:29:16<11:27:56, 11.21s/it][2022-12-20 14:51:52,267] [INFO] [timer.py:197:stop] 0/2672, RunningAvgSamplesPerSec=5.867263905428857, CurrSamplesPerSec=5.344410862900042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1320/5000 [4:29:27<11:24:28, 11.16s/it][2022-12-20 14:52:03,271] [INFO] [timer.py:197:stop] 0/2674, RunningAvgSamplesPerSec=5.867274408940137, CurrSamplesPerSec=5.341787366359527, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1321/5000 [4:29:38<11:21:25, 11.11s/it][2022-12-20 14:52:14,367] [INFO] [timer.py:197:stop] 0/2676, RunningAvgSamplesPerSec=5.867248295502644, CurrSamplesPerSec=5.293153795813945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1322/5000 [4:29:49<11:20:54, 11.11s/it][2022-12-20 14:52:25,380] [INFO] [timer.py:197:stop] 0/2678, RunningAvgSamplesPerSec=5.867256981804069, CurrSamplesPerSec=5.351379640283278, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1323/5000 [4:30:00<11:18:59, 11.08s/it][2022-12-20 14:52:36,369] [INFO] [logging.py:68:log_dist] [Rank 0] step=1340, skipped=3, lr=[8.142222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:52:36,371] [INFO] [timer.py:197:stop] 0/2680, RunningAvgSamplesPerSec=5.8672728814667785, CurrSamplesPerSec=5.357554904990007, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1324/5000 [4:30:11<11:17:10, 11.05s/it][2022-12-20 14:52:47,384] [INFO] [timer.py:197:stop] 0/2682, RunningAvgSamplesPerSec=5.867279948228556, CurrSamplesPerSec=5.329852787625319, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 26%|██▋       | 1325/5000 [4:30:22<11:16:15, 11.04s/it]
                                                        
{'loss': 0.0006, 'learning_rate': 8.14e-06, 'epoch': 32.31}

 26%|██▋       | 1325/5000 [4:30:22<11:16:15, 11.04s/it][2022-12-20 14:52:58,648] [INFO] [timer.py:197:stop] 0/2684, RunningAvgSamplesPerSec=5.8671873148079445, CurrSamplesPerSec=5.089257000264098, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1326/5000 [4:30:34<11:20:10, 11.11s/it][2022-12-20 14:53:09,716] [INFO] [timer.py:197:stop] 0/2686, RunningAvgSamplesPerSec=5.86717231435457, CurrSamplesPerSec=5.304238226057561, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1327/5000 [4:30:45<11:19:15, 11.10s/it][2022-12-20 14:53:20,715] [INFO] [timer.py:197:stop] 0/2688, RunningAvgSamplesPerSec=5.8671902036318535, CurrSamplesPerSec=5.3446607122320975, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1328/5000 [4:30:56<11:17:17, 11.07s/it][2022-12-20 14:53:31,724] [INFO] [timer.py:197:stop] 0/2690, RunningAvgSamplesPerSec=5.867198853654752, CurrSamplesPerSec=5.319216997369674, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1329/5000 [4:31:07<11:16:02, 11.05s/it][2022-12-20 14:53:42,774] [INFO] [timer.py:197:stop] 0/2692, RunningAvgSamplesPerSec=5.8671912004198425, CurrSamplesPerSec=5.315058915308872, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1330/5000 [4:31:18<11:15:52, 11.05s/it][2022-12-20 14:53:53,796] [INFO] [timer.py:197:stop] 0/2694, RunningAvgSamplesPerSec=5.867194948034198, CurrSamplesPerSec=5.326780116065605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1331/5000 [4:31:29<11:15:10, 11.04s/it][2022-12-20 14:54:04,945] [INFO] [timer.py:197:stop] 0/2696, RunningAvgSamplesPerSec=5.867148181595067, CurrSamplesPerSec=5.229207053416619, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1332/5000 [4:31:40<11:16:58, 11.07s/it][2022-12-20 14:54:15,953] [INFO] [timer.py:197:stop] 0/2698, RunningAvgSamplesPerSec=5.867157803444068, CurrSamplesPerSec=5.320486781644827, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1333/5000 [4:31:51<11:15:36, 11.05s/it][2022-12-20 14:54:27,049] [INFO] [logging.py:68:log_dist] [Rank 0] step=1350, skipped=3, lr=[8.120000000000002e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:54:27,050] [INFO] [timer.py:197:stop] 0/2700, RunningAvgSamplesPerSec=5.867140936807669, CurrSamplesPerSec=5.2963597860505915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1334/5000 [4:32:02<11:16:10, 11.07s/it][2022-12-20 14:54:38,159] [INFO] [timer.py:197:stop] 0/2702, RunningAvgSamplesPerSec=5.867110763849676, CurrSamplesPerSec=5.302708638765092, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1335/5000 [4:32:13<11:16:45, 11.08s/it][2022-12-20 14:54:49,144] [INFO] [timer.py:197:stop] 0/2704, RunningAvgSamplesPerSec=5.867128856162109, CurrSamplesPerSec=5.34210862431608, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1336/5000 [4:32:24<11:14:52, 11.05s/it][2022-12-20 14:55:00,186] [INFO] [timer.py:197:stop] 0/2706, RunningAvgSamplesPerSec=5.86713148198925, CurrSamplesPerSec=5.332181543554038, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1337/5000 [4:32:35<11:14:29, 11.05s/it][2022-12-20 14:55:11,280] [INFO] [timer.py:197:stop] 0/2708, RunningAvgSamplesPerSec=5.867106555650406, CurrSamplesPerSec=5.200583008320454, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1338/5000 [4:32:46<11:15:08, 11.06s/it][2022-12-20 14:55:22,304] [INFO] [timer.py:197:stop] 0/2710, RunningAvgSamplesPerSec=5.867109746674261, CurrSamplesPerSec=5.346511473663643, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1339/5000 [4:32:57<11:14:15, 11.05s/it][2022-12-20 14:55:33,336] [INFO] [timer.py:197:stop] 0/2712, RunningAvgSamplesPerSec=5.867109268480701, CurrSamplesPerSec=5.303738744676808, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1340/5000 [4:33:08<11:13:45, 11.05s/it][2022-12-20 14:55:44,573] [INFO] [timer.py:197:stop] 0/2714, RunningAvgSamplesPerSec=5.867028065655926, CurrSamplesPerSec=5.140866532197013, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1341/5000 [4:33:20<11:17:04, 11.10s/it][2022-12-20 14:55:55,571] [INFO] [timer.py:197:stop] 0/2716, RunningAvgSamplesPerSec=5.867041013729103, CurrSamplesPerSec=5.32446325629723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1342/5000 [4:33:31<11:14:58, 11.07s/it][2022-12-20 14:56:06,567] [INFO] [timer.py:197:stop] 0/2718, RunningAvgSamplesPerSec=5.867055199909407, CurrSamplesPerSec=5.319817865885367, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1343/5000 [4:33:42<11:13:25, 11.05s/it][2022-12-20 14:56:17,623] [INFO] [logging.py:68:log_dist] [Rank 0] step=1360, skipped=3, lr=[8.09777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:56:17,625] [INFO] [timer.py:197:stop] 0/2720, RunningAvgSamplesPerSec=5.867044344601528, CurrSamplesPerSec=5.28937692906708, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1344/5000 [4:33:53<11:13:23, 11.05s/it][2022-12-20 14:56:28,670] [INFO] [timer.py:197:stop] 0/2722, RunningAvgSamplesPerSec=5.867039050593108, CurrSamplesPerSec=5.329665694614278, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1345/5000 [4:34:04<11:13:05, 11.05s/it][2022-12-20 14:56:39,681] [INFO] [timer.py:197:stop] 0/2724, RunningAvgSamplesPerSec=5.8670470465713445, CurrSamplesPerSec=5.314920003595611, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1346/5000 [4:34:15<11:12:13, 11.04s/it][2022-12-20 14:56:50,801] [INFO] [timer.py:197:stop] 0/2726, RunningAvgSamplesPerSec=5.867012136388272, CurrSamplesPerSec=5.2200764224035066, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1347/5000 [4:34:26<11:13:31, 11.06s/it][2022-12-20 14:57:01,822] [INFO] [timer.py:197:stop] 0/2728, RunningAvgSamplesPerSec=5.867016487729158, CurrSamplesPerSec=5.305950129628746, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1348/5000 [4:34:37<11:12:35, 11.05s/it][2022-12-20 14:57:12,815] [INFO] [timer.py:197:stop] 0/2730, RunningAvgSamplesPerSec=5.867037933345068, CurrSamplesPerSec=5.365565542459984, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1349/5000 [4:34:48<11:11:21, 11.03s/it][2022-12-20 14:57:24,036] [INFO] [timer.py:197:stop] 0/2732, RunningAvgSamplesPerSec=5.866963522194716, CurrSamplesPerSec=5.153858568615388, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1350/5000 [4:34:59<11:14:36, 11.09s/it]
                                                        
{'loss': 0.0006, 'learning_rate': 8.084444444444444e-06, 'epoch': 32.92}

 27%|██▋       | 1350/5000 [4:34:59<11:14:36, 11.09s/it][2022-12-20 14:57:35,075] [INFO] [timer.py:197:stop] 0/2734, RunningAvgSamplesPerSec=5.866960909313108, CurrSamplesPerSec=5.326678219771939, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1351/5000 [4:35:10<11:13:29, 11.07s/it][2022-12-20 14:57:46,059] [INFO] [timer.py:197:stop] 0/2736, RunningAvgSamplesPerSec=5.866984988698939, CurrSamplesPerSec=5.358850763780918, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1352/5000 [4:35:21<11:11:40, 11.05s/it][2022-12-20 14:57:57,176] [INFO] [timer.py:197:stop] 0/2738, RunningAvgSamplesPerSec=5.866951168860731, CurrSamplesPerSec=5.241161881620681, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1353/5000 [4:35:32<11:12:45, 11.07s/it][2022-12-20 14:58:07,277] [INFO] [logging.py:68:log_dist] [Rank 0] step=1370, skipped=3, lr=[8.075555555555557e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:58:07,279] [INFO] [timer.py:197:stop] 0/2740, RunningAvgSamplesPerSec=5.867311835449159, CurrSamplesPerSec=5.319994568138901, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1354/5000 [4:35:47<12:24:49, 12.26s/it][2022-12-20 14:58:18,278] [INFO] [timer.py:197:stop] 0/2742, RunningAvgSamplesPerSec=5.867323816245555, CurrSamplesPerSec=5.3269276823392335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1355/5000 [4:35:58<12:01:21, 11.87s/it][2022-12-20 14:58:29,399] [INFO] [timer.py:197:stop] 0/2744, RunningAvgSamplesPerSec=5.867288330204955, CurrSamplesPerSec=5.2052450615555275, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1356/5000 [4:36:09<11:47:16, 11.65s/it][2022-12-20 14:58:40,344] [INFO] [timer.py:197:stop] 0/2746, RunningAvgSamplesPerSec=5.867321968544436, CurrSamplesPerSec=5.350604813552862, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1357/5000 [4:36:20<11:34:48, 11.44s/it][2022-12-20 14:58:51,315] [INFO] [timer.py:197:stop] 0/2748, RunningAvgSamplesPerSec=5.867345307267528, CurrSamplesPerSec=5.351716990418064, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1358/5000 [4:36:31<11:26:41, 11.31s/it][2022-12-20 14:59:02,492] [INFO] [timer.py:197:stop] 0/2750, RunningAvgSamplesPerSec=5.867287941720961, CurrSamplesPerSec=5.205512553479383, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1359/5000 [4:36:43<11:23:47, 11.27s/it][2022-12-20 14:59:13,476] [INFO] [timer.py:197:stop] 0/2752, RunningAvgSamplesPerSec=5.867306221692765, CurrSamplesPerSec=5.362882455469781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1360/5000 [4:36:54<11:18:29, 11.18s/it][2022-12-20 14:59:24,499] [INFO] [timer.py:197:stop] 0/2754, RunningAvgSamplesPerSec=5.867309435728499, CurrSamplesPerSec=5.330072490530641, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1361/5000 [4:37:05<11:15:30, 11.14s/it][2022-12-20 14:59:35,577] [INFO] [timer.py:197:stop] 0/2756, RunningAvgSamplesPerSec=5.867290636310909, CurrSamplesPerSec=5.288418655675564, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1362/5000 [4:37:16<11:14:10, 11.12s/it][2022-12-20 14:59:46,567] [INFO] [timer.py:197:stop] 0/2758, RunningAvgSamplesPerSec=5.86730639472436, CurrSamplesPerSec=5.362812386012836, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1363/5000 [4:37:27<11:11:08, 11.07s/it][2022-12-20 14:59:57,579] [INFO] [logging.py:68:log_dist] [Rank 0] step=1380, skipped=3, lr=[8.053333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-20 14:59:57,580] [INFO] [timer.py:197:stop] 0/2760, RunningAvgSamplesPerSec=5.867312972266146, CurrSamplesPerSec=5.31772321523023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1364/5000 [4:37:38<11:09:45, 11.05s/it][2022-12-20 15:00:08,673] [INFO] [timer.py:197:stop] 0/2762, RunningAvgSamplesPerSec=5.867288437756883, CurrSamplesPerSec=5.242430297776019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1365/5000 [4:37:49<11:11:06, 11.08s/it][2022-12-20 15:00:19,732] [INFO] [timer.py:197:stop] 0/2764, RunningAvgSamplesPerSec=5.867277296831097, CurrSamplesPerSec=5.309571326943358, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1366/5000 [4:38:00<11:10:08, 11.06s/it][2022-12-20 15:00:30,740] [INFO] [timer.py:197:stop] 0/2766, RunningAvgSamplesPerSec=5.867286157355487, CurrSamplesPerSec=5.332981344778789, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1367/5000 [4:38:11<11:08:55, 11.05s/it][2022-12-20 15:00:41,891] [INFO] [timer.py:197:stop] 0/2768, RunningAvgSamplesPerSec=5.867239115212334, CurrSamplesPerSec=5.2082219278702215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1368/5000 [4:38:22<11:10:39, 11.08s/it][2022-12-20 15:00:52,914] [INFO] [timer.py:197:stop] 0/2770, RunningAvgSamplesPerSec=5.867241686279702, CurrSamplesPerSec=5.320572622546393, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1369/5000 [4:38:33<11:09:32, 11.06s/it][2022-12-20 15:01:03,909] [INFO] [timer.py:197:stop] 0/2772, RunningAvgSamplesPerSec=5.867255286219318, CurrSamplesPerSec=5.350025333528226, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1370/5000 [4:38:44<11:07:59, 11.04s/it][2022-12-20 15:01:15,033] [INFO] [timer.py:197:stop] 0/2774, RunningAvgSamplesPerSec=5.8672187708264465, CurrSamplesPerSec=5.23199255050194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1371/5000 [4:38:55<11:09:39, 11.07s/it][2022-12-20 15:01:26,029] [INFO] [timer.py:197:stop] 0/2776, RunningAvgSamplesPerSec=5.867232050620762, CurrSamplesPerSec=5.361408811306173, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1372/5000 [4:39:06<11:08:02, 11.05s/it][2022-12-20 15:01:37,050] [INFO] [timer.py:197:stop] 0/2778, RunningAvgSamplesPerSec=5.867235770852498, CurrSamplesPerSec=5.336036355951057, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1373/5000 [4:39:17<11:07:17, 11.04s/it][2022-12-20 15:01:48,176] [INFO] [logging.py:68:log_dist] [Rank 0] step=1390, skipped=3, lr=[8.031111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:01:48,179] [INFO] [timer.py:197:stop] 0/2780, RunningAvgSamplesPerSec=5.867197504425831, CurrSamplesPerSec=5.2383582328035745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 27%|██▋       | 1374/5000 [4:39:28<11:08:35, 11.06s/it][2022-12-20 15:01:59,202] [INFO] [timer.py:197:stop] 0/2782, RunningAvgSamplesPerSec=5.867200010984818, CurrSamplesPerSec=5.322838600251385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1375/5000 [4:39:39<11:07:57, 11.06s/it]
                                                        
{'loss': 0.0006, 'learning_rate': 8.02888888888889e-06, 'epoch': 33.53}

 28%|██▊       | 1375/5000 [4:39:39<11:07:57, 11.06s/it][2022-12-20 15:02:10,247] [INFO] [timer.py:197:stop] 0/2784, RunningAvgSamplesPerSec=5.867201221496921, CurrSamplesPerSec=5.318055913754904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1376/5000 [4:39:50<11:09:46, 11.09s/it][2022-12-20 15:02:21,549] [INFO] [timer.py:197:stop] 0/2786, RunningAvgSamplesPerSec=5.867155904526517, CurrSamplesPerSec=5.314694813669712, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1377/5000 [4:40:02<11:10:54, 11.11s/it][2022-12-20 15:02:32,588] [INFO] [timer.py:197:stop] 0/2788, RunningAvgSamplesPerSec=5.867152144509463, CurrSamplesPerSec=5.304910358263791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1378/5000 [4:40:13<11:09:54, 11.10s/it][2022-12-20 15:02:43,655] [INFO] [timer.py:197:stop] 0/2790, RunningAvgSamplesPerSec=5.867141676761146, CurrSamplesPerSec=5.304385803813207, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1379/5000 [4:40:24<11:07:58, 11.07s/it][2022-12-20 15:02:54,752] [INFO] [timer.py:197:stop] 0/2792, RunningAvgSamplesPerSec=5.86711616600331, CurrSamplesPerSec=5.221499997160066, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1380/5000 [4:40:35<11:09:38, 11.10s/it][2022-12-20 15:03:05,842] [INFO] [timer.py:197:stop] 0/2794, RunningAvgSamplesPerSec=5.86709356103946, CurrSamplesPerSec=5.291232773241094, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1381/5000 [4:40:46<11:08:44, 11.09s/it][2022-12-20 15:03:16,873] [INFO] [timer.py:197:stop] 0/2796, RunningAvgSamplesPerSec=5.867099498805753, CurrSamplesPerSec=5.3161496245381015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1382/5000 [4:40:57<11:06:48, 11.06s/it][2022-12-20 15:03:27,880] [INFO] [timer.py:197:stop] 0/2798, RunningAvgSamplesPerSec=5.867108097276354, CurrSamplesPerSec=5.301269130778188, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1383/5000 [4:41:08<11:06:03, 11.05s/it][2022-12-20 15:03:38,895] [INFO] [logging.py:68:log_dist] [Rank 0] step=1400, skipped=3, lr=[8.00888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:03:38,897] [INFO] [timer.py:197:stop] 0/2800, RunningAvgSamplesPerSec=5.867113614161458, CurrSamplesPerSec=5.310326750466165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1384/5000 [4:41:19<11:05:05, 11.04s/it][2022-12-20 15:03:49,905] [INFO] [timer.py:197:stop] 0/2802, RunningAvgSamplesPerSec=5.867122233358075, CurrSamplesPerSec=5.307900537545747, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1385/5000 [4:41:30<11:04:34, 11.03s/it][2022-12-20 15:04:01,225] [INFO] [timer.py:197:stop] 0/2804, RunningAvgSamplesPerSec=5.867011169317707, CurrSamplesPerSec=5.053979424736549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1386/5000 [4:41:41<11:10:13, 11.13s/it][2022-12-20 15:04:12,234] [INFO] [timer.py:197:stop] 0/2806, RunningAvgSamplesPerSec=5.867019343469579, CurrSamplesPerSec=5.344345531365432, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1387/5000 [4:41:52<11:06:56, 11.08s/it][2022-12-20 15:04:23,199] [INFO] [timer.py:197:stop] 0/2808, RunningAvgSamplesPerSec=5.8670449305129555, CurrSamplesPerSec=5.337076056728726, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1388/5000 [4:42:03<11:05:42, 11.06s/it][2022-12-20 15:04:34,226] [INFO] [timer.py:197:stop] 0/2810, RunningAvgSamplesPerSec=5.867045932593934, CurrSamplesPerSec=5.32805562930396, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1389/5000 [4:42:14<11:04:25, 11.04s/it][2022-12-20 15:04:45,236] [INFO] [timer.py:197:stop] 0/2812, RunningAvgSamplesPerSec=5.867059625862733, CurrSamplesPerSec=5.316675876531561, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1390/5000 [4:42:25<11:04:24, 11.04s/it][2022-12-20 15:04:56,265] [INFO] [timer.py:197:stop] 0/2814, RunningAvgSamplesPerSec=5.867059887804062, CurrSamplesPerSec=5.334593328744016, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1391/5000 [4:42:36<11:03:33, 11.03s/it][2022-12-20 15:05:07,324] [INFO] [timer.py:197:stop] 0/2816, RunningAvgSamplesPerSec=5.8670490291784665, CurrSamplesPerSec=5.2889638160500185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1392/5000 [4:42:47<11:04:29, 11.05s/it][2022-12-20 15:05:18,400] [INFO] [timer.py:197:stop] 0/2818, RunningAvgSamplesPerSec=5.867034415730452, CurrSamplesPerSec=5.3041747114546505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1393/5000 [4:42:58<11:03:26, 11.04s/it][2022-12-20 15:05:29,401] [INFO] [logging.py:68:log_dist] [Rank 0] step=1410, skipped=3, lr=[7.986666666666668e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:05:29,403] [INFO] [timer.py:197:stop] 0/2820, RunningAvgSamplesPerSec=5.867045058085155, CurrSamplesPerSec=5.303396938163012, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1394/5000 [4:43:09<11:03:27, 11.04s/it][2022-12-20 15:05:39,642] [INFO] [timer.py:197:stop] 0/2822, RunningAvgSamplesPerSec=5.86734343328232, CurrSamplesPerSec=6.111975174997072, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 15:05:50,699] [INFO] [timer.py:197:stop] 0/2824, RunningAvgSamplesPerSec=5.867338862693114, CurrSamplesPerSec=5.323293336434344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1395/5000 [4:43:26<12:38:56, 12.63s/it][2022-12-20 15:06:01,797] [INFO] [timer.py:197:stop] 0/2826, RunningAvgSamplesPerSec=5.867326226765326, CurrSamplesPerSec=5.294675786349785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1396/5000 [4:43:37<12:11:05, 12.17s/it][2022-12-20 15:06:13,180] [INFO] [timer.py:197:stop] 0/2828, RunningAvgSamplesPerSec=5.867192010740171, CurrSamplesPerSec=4.986640689031509, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1397/5000 [4:43:48<11:56:41, 11.93s/it][2022-12-20 15:06:24,194] [INFO] [timer.py:197:stop] 0/2830, RunningAvgSamplesPerSec=5.867198645599095, CurrSamplesPerSec=5.347763641102168, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1398/5000 [4:43:59<11:39:54, 11.66s/it][2022-12-20 15:06:35,232] [INFO] [timer.py:197:stop] 0/2832, RunningAvgSamplesPerSec=5.867196186659145, CurrSamplesPerSec=5.319893985584041, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1399/5000 [4:44:10<11:28:32, 11.47s/it][2022-12-20 15:06:46,411] [INFO] [timer.py:197:stop] 0/2834, RunningAvgSamplesPerSec=5.867140617621161, CurrSamplesPerSec=5.204313394928186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1400/5000 [4:44:22<11:23:04, 11.38s/it]
                                                        
{'loss': 0.001, 'learning_rate': 7.971111111111111e-06, 'epoch': 34.14}

 28%|██▊       | 1400/5000 [4:44:22<11:23:04, 11.38s/it][2022-12-20 15:06:57,438] [INFO] [timer.py:197:stop] 0/2836, RunningAvgSamplesPerSec=5.8671491837554175, CurrSamplesPerSec=5.335715191541583, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1401/5000 [4:44:33<11:16:26, 11.28s/it][2022-12-20 15:07:08,436] [INFO] [timer.py:197:stop] 0/2838, RunningAvgSamplesPerSec=5.86716195502852, CurrSamplesPerSec=5.354130892406157, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1402/5000 [4:44:44<11:11:13, 11.19s/it][2022-12-20 15:07:19,529] [INFO] [logging.py:68:log_dist] [Rank 0] step=1420, skipped=3, lr=[7.964444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:07:19,530] [INFO] [timer.py:197:stop] 0/2840, RunningAvgSamplesPerSec=5.8671376598872245, CurrSamplesPerSec=5.274742328287568, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1403/5000 [4:44:55<11:09:15, 11.16s/it][2022-12-20 15:07:30,626] [INFO] [timer.py:197:stop] 0/2842, RunningAvgSamplesPerSec=5.8671191455769565, CurrSamplesPerSec=5.304923777484285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1404/5000 [4:45:06<11:07:51, 11.14s/it][2022-12-20 15:07:41,682] [INFO] [timer.py:197:stop] 0/2844, RunningAvgSamplesPerSec=5.867115892199886, CurrSamplesPerSec=5.310166656676801, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1405/5000 [4:45:17<11:06:06, 11.12s/it][2022-12-20 15:07:52,865] [INFO] [timer.py:197:stop] 0/2846, RunningAvgSamplesPerSec=5.867058747024386, CurrSamplesPerSec=5.191834732693543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1406/5000 [4:45:28<11:07:05, 11.14s/it][2022-12-20 15:08:03,883] [INFO] [timer.py:197:stop] 0/2848, RunningAvgSamplesPerSec=5.8670640955115605, CurrSamplesPerSec=5.324719693711162, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1407/5000 [4:45:39<11:04:46, 11.10s/it][2022-12-20 15:08:14,866] [INFO] [timer.py:197:stop] 0/2850, RunningAvgSamplesPerSec=5.867088915224344, CurrSamplesPerSec=5.345768289960463, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1408/5000 [4:45:50<11:02:27, 11.07s/it][2022-12-20 15:08:26,189] [INFO] [timer.py:197:stop] 0/2852, RunningAvgSamplesPerSec=5.866979385388325, CurrSamplesPerSec=5.087858901602813, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1409/5000 [4:46:01<11:06:55, 11.14s/it][2022-12-20 15:08:37,162] [INFO] [timer.py:197:stop] 0/2854, RunningAvgSamplesPerSec=5.8670037442008995, CurrSamplesPerSec=5.337148214263844, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1410/5000 [4:46:12<11:03:40, 11.09s/it][2022-12-20 15:08:48,142] [INFO] [timer.py:197:stop] 0/2856, RunningAvgSamplesPerSec=5.867023199587913, CurrSamplesPerSec=5.344922291630221, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1411/5000 [4:46:23<11:01:28, 11.06s/it][2022-12-20 15:08:59,167] [INFO] [timer.py:197:stop] 0/2858, RunningAvgSamplesPerSec=5.867025529500357, CurrSamplesPerSec=5.28827196504118, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1412/5000 [4:46:34<11:00:42, 11.05s/it][2022-12-20 15:09:10,134] [INFO] [logging.py:68:log_dist] [Rank 0] step=1430, skipped=3, lr=[7.942222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:09:10,135] [INFO] [timer.py:197:stop] 0/2860, RunningAvgSamplesPerSec=5.867055039669358, CurrSamplesPerSec=5.364055265651521, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1413/5000 [4:46:45<10:59:03, 11.02s/it][2022-12-20 15:09:21,147] [INFO] [timer.py:197:stop] 0/2862, RunningAvgSamplesPerSec=5.867061760960472, CurrSamplesPerSec=5.3138590418611855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1414/5000 [4:46:56<10:58:39, 11.02s/it][2022-12-20 15:09:32,507] [INFO] [timer.py:197:stop] 0/2864, RunningAvgSamplesPerSec=5.866937931910629, CurrSamplesPerSec=5.042016311736068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1415/5000 [4:47:08<11:04:34, 11.12s/it][2022-12-20 15:09:43,451] [INFO] [timer.py:197:stop] 0/2866, RunningAvgSamplesPerSec=5.866971301400676, CurrSamplesPerSec=5.345289908230867, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1416/5000 [4:47:19<11:01:10, 11.07s/it][2022-12-20 15:09:54,511] [INFO] [timer.py:197:stop] 0/2868, RunningAvgSamplesPerSec=5.866960266775052, CurrSamplesPerSec=5.311099202067241, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1417/5000 [4:47:30<11:00:50, 11.07s/it][2022-12-20 15:10:05,475] [INFO] [timer.py:197:stop] 0/2870, RunningAvgSamplesPerSec=5.866985461643725, CurrSamplesPerSec=5.36118351947005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1418/5000 [4:47:41<10:58:49, 11.04s/it][2022-12-20 15:10:16,514] [INFO] [timer.py:197:stop] 0/2872, RunningAvgSamplesPerSec=5.866982604336558, CurrSamplesPerSec=5.3334467170235955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1419/5000 [4:47:52<10:58:41, 11.04s/it][2022-12-20 15:10:27,548] [INFO] [timer.py:197:stop] 0/2874, RunningAvgSamplesPerSec=5.86698156694766, CurrSamplesPerSec=5.3236098398253295, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1420/5000 [4:48:03<10:58:28, 11.04s/it][2022-12-20 15:10:38,915] [INFO] [timer.py:197:stop] 0/2876, RunningAvgSamplesPerSec=5.866856104918715, CurrSamplesPerSec=5.0569619781005954, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1421/5000 [4:48:14<11:04:12, 11.14s/it][2022-12-20 15:10:49,925] [INFO] [timer.py:197:stop] 0/2878, RunningAvgSamplesPerSec=5.86686484900348, CurrSamplesPerSec=5.306912039785037, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1422/5000 [4:48:25<11:01:46, 11.10s/it][2022-12-20 15:11:00,946] [INFO] [logging.py:68:log_dist] [Rank 0] step=1440, skipped=3, lr=[7.92e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:11:00,949] [INFO] [timer.py:197:stop] 0/2880, RunningAvgSamplesPerSec=5.866874588050978, CurrSamplesPerSec=5.351646785652113, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1423/5000 [4:48:36<11:00:16, 11.08s/it][2022-12-20 15:11:12,008] [INFO] [timer.py:197:stop] 0/2882, RunningAvgSamplesPerSec=5.866864039138834, CurrSamplesPerSec=5.29073219265182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1424/5000 [4:48:47<10:59:48, 11.07s/it][2022-12-20 15:11:22,989] [INFO] [timer.py:197:stop] 0/2884, RunningAvgSamplesPerSec=5.866890220820885, CurrSamplesPerSec=5.378103806853964, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 28%|██▊       | 1425/5000 [4:48:58<10:58:00, 11.04s/it]
                                                        
{'loss': 0.0009, 'learning_rate': 7.915555555555557e-06, 'epoch': 34.75}

 28%|██▊       | 1425/5000 [4:48:58<10:58:00, 11.04s/it][2022-12-20 15:11:34,049] [INFO] [timer.py:197:stop] 0/2886, RunningAvgSamplesPerSec=5.866885264751231, CurrSamplesPerSec=5.279103309733641, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1426/5000 [4:49:09<10:58:08, 11.05s/it][2022-12-20 15:11:45,134] [INFO] [timer.py:197:stop] 0/2888, RunningAvgSamplesPerSec=5.866865850099601, CurrSamplesPerSec=5.279858188051182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1427/5000 [4:49:20<10:58:35, 11.06s/it][2022-12-20 15:11:56,172] [INFO] [timer.py:197:stop] 0/2890, RunningAvgSamplesPerSec=5.866863865739063, CurrSamplesPerSec=5.298604345623919, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1428/5000 [4:49:31<10:58:01, 11.05s/it][2022-12-20 15:12:07,200] [INFO] [timer.py:197:stop] 0/2892, RunningAvgSamplesPerSec=5.866866299764048, CurrSamplesPerSec=5.294540653151145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1429/5000 [4:49:42<10:57:24, 11.05s/it][2022-12-20 15:12:18,281] [INFO] [timer.py:197:stop] 0/2894, RunningAvgSamplesPerSec=5.866848617162379, CurrSamplesPerSec=5.292986177705614, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1430/5000 [4:49:53<10:57:50, 11.06s/it][2022-12-20 15:12:29,248] [INFO] [timer.py:197:stop] 0/2896, RunningAvgSamplesPerSec=5.86687662329632, CurrSamplesPerSec=5.352104750222908, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1431/5000 [4:50:04<10:56:04, 11.03s/it][2022-12-20 15:12:40,294] [INFO] [timer.py:197:stop] 0/2898, RunningAvgSamplesPerSec=5.8668784047420335, CurrSamplesPerSec=5.335385157964976, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1432/5000 [4:50:15<10:56:10, 11.03s/it][2022-12-20 15:12:51,273] [INFO] [logging.py:68:log_dist] [Rank 0] step=1450, skipped=3, lr=[7.897777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:12:51,280] [INFO] [timer.py:197:stop] 0/2900, RunningAvgSamplesPerSec=5.866895486059719, CurrSamplesPerSec=5.357797429754562, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1433/5000 [4:50:26<10:55:19, 11.02s/it][2022-12-20 15:13:02,369] [INFO] [timer.py:197:stop] 0/2902, RunningAvgSamplesPerSec=5.866903808135881, CurrSamplesPerSec=5.349864116651812, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1434/5000 [4:50:37<10:56:07, 11.04s/it][2022-12-20 15:13:13,428] [INFO] [timer.py:197:stop] 0/2904, RunningAvgSamplesPerSec=5.866893323105897, CurrSamplesPerSec=5.3015800887893425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1435/5000 [4:50:49<10:56:16, 11.05s/it][2022-12-20 15:13:23,529] [INFO] [timer.py:197:stop] 0/2906, RunningAvgSamplesPerSec=5.867233968061934, CurrSamplesPerSec=5.316965054129023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1436/5000 [4:51:04<12:08:58, 12.27s/it][2022-12-20 15:13:34,646] [INFO] [timer.py:197:stop] 0/2908, RunningAvgSamplesPerSec=5.867201605389572, CurrSamplesPerSec=5.315080173651068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▊       | 1437/5000 [4:51:15<11:47:16, 11.91s/it][2022-12-20 15:13:45,694] [INFO] [timer.py:197:stop] 0/2910, RunningAvgSamplesPerSec=5.867195205222465, CurrSamplesPerSec=5.331888166698414, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1438/5000 [4:51:26<11:30:19, 11.63s/it][2022-12-20 15:13:56,653] [INFO] [timer.py:197:stop] 0/2912, RunningAvgSamplesPerSec=5.8672217783593545, CurrSamplesPerSec=5.341562232422383, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1439/5000 [4:51:37<11:19:11, 11.44s/it][2022-12-20 15:14:07,698] [INFO] [timer.py:197:stop] 0/2914, RunningAvgSamplesPerSec=5.867216504522773, CurrSamplesPerSec=5.313482693735088, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1440/5000 [4:51:48<11:12:02, 11.33s/it][2022-12-20 15:14:18,760] [INFO] [timer.py:197:stop] 0/2916, RunningAvgSamplesPerSec=5.867204581431818, CurrSamplesPerSec=5.30529115501448, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1441/5000 [4:51:59<11:07:34, 11.25s/it][2022-12-20 15:14:29,825] [INFO] [timer.py:197:stop] 0/2918, RunningAvgSamplesPerSec=5.867191804776315, CurrSamplesPerSec=5.3240522475203385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1442/5000 [4:52:10<11:02:55, 11.18s/it][2022-12-20 15:14:40,834] [INFO] [logging.py:68:log_dist] [Rank 0] step=1460, skipped=3, lr=[7.875555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:14:40,835] [INFO] [timer.py:197:stop] 0/2920, RunningAvgSamplesPerSec=5.867199079372331, CurrSamplesPerSec=5.317994385621187, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1443/5000 [4:52:21<11:00:33, 11.14s/it][2022-12-20 15:14:51,963] [INFO] [timer.py:197:stop] 0/2922, RunningAvgSamplesPerSec=5.867163609735234, CurrSamplesPerSec=5.256612283581605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1444/5000 [4:52:32<10:59:39, 11.13s/it][2022-12-20 15:15:03,026] [INFO] [timer.py:197:stop] 0/2924, RunningAvgSamplesPerSec=5.867151792571252, CurrSamplesPerSec=5.290244634796273, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1445/5000 [4:52:43<10:57:50, 11.10s/it][2022-12-20 15:15:14,034] [INFO] [timer.py:197:stop] 0/2926, RunningAvgSamplesPerSec=5.867160064009995, CurrSamplesPerSec=5.317455021837314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1446/5000 [4:52:54<10:56:44, 11.09s/it][2022-12-20 15:15:25,081] [INFO] [timer.py:197:stop] 0/2928, RunningAvgSamplesPerSec=5.867154230800252, CurrSamplesPerSec=5.320977187718031, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1447/5000 [4:53:05<10:55:26, 11.07s/it][2022-12-20 15:15:36,091] [INFO] [timer.py:197:stop] 0/2930, RunningAvgSamplesPerSec=5.86716221353412, CurrSamplesPerSec=5.334001836847998, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1448/5000 [4:53:16<10:53:28, 11.04s/it][2022-12-20 15:15:47,122] [INFO] [timer.py:197:stop] 0/2932, RunningAvgSamplesPerSec=5.867167395489543, CurrSamplesPerSec=5.2794615125578925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1449/5000 [4:53:27<10:54:32, 11.06s/it][2022-12-20 15:15:58,239] [INFO] [timer.py:197:stop] 0/2934, RunningAvgSamplesPerSec=5.86714863172802, CurrSamplesPerSec=5.287083945794458, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1450/5000 [4:53:38<10:55:03, 11.07s/it]
                                                        
{'loss': 0.0007, 'learning_rate': 7.860000000000001e-06, 'epoch': 35.36}

 29%|██▉       | 1450/5000 [4:53:38<10:55:03, 11.07s/it][2022-12-20 15:16:09,321] [INFO] [timer.py:197:stop] 0/2936, RunningAvgSamplesPerSec=5.867140142287413, CurrSamplesPerSec=5.299458760613213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1451/5000 [4:53:49<10:54:30, 11.07s/it][2022-12-20 15:16:20,278] [INFO] [timer.py:197:stop] 0/2938, RunningAvgSamplesPerSec=5.867171443541817, CurrSamplesPerSec=5.370010380915503, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1452/5000 [4:54:00<10:52:02, 11.03s/it][2022-12-20 15:16:31,293] [INFO] [logging.py:68:log_dist] [Rank 0] step=1470, skipped=3, lr=[7.853333333333333e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:16:31,294] [INFO] [timer.py:197:stop] 0/2940, RunningAvgSamplesPerSec=5.867176768975583, CurrSamplesPerSec=5.298791983678718, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1453/5000 [4:54:11<10:51:30, 11.02s/it][2022-12-20 15:16:42,288] [INFO] [timer.py:197:stop] 0/2942, RunningAvgSamplesPerSec=5.86719038388301, CurrSamplesPerSec=5.3109327571587315, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1454/5000 [4:54:22<10:50:51, 11.01s/it][2022-12-20 15:16:53,330] [INFO] [timer.py:197:stop] 0/2944, RunningAvgSamplesPerSec=5.867186414520064, CurrSamplesPerSec=5.269410659324282, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1455/5000 [4:54:33<10:51:32, 11.03s/it][2022-12-20 15:17:04,313] [INFO] [timer.py:197:stop] 0/2946, RunningAvgSamplesPerSec=5.8672037531643735, CurrSamplesPerSec=5.33894004454204, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1456/5000 [4:54:44<10:50:57, 11.02s/it][2022-12-20 15:17:15,345] [INFO] [timer.py:197:stop] 0/2948, RunningAvgSamplesPerSec=5.867203384647868, CurrSamplesPerSec=5.314893063982781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1457/5000 [4:54:55<10:51:28, 11.03s/it][2022-12-20 15:17:26,449] [INFO] [timer.py:197:stop] 0/2950, RunningAvgSamplesPerSec=5.867176306276565, CurrSamplesPerSec=5.275464441914845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1458/5000 [4:55:06<10:51:18, 11.03s/it][2022-12-20 15:17:37,388] [INFO] [timer.py:197:stop] 0/2952, RunningAvgSamplesPerSec=5.867209703867827, CurrSamplesPerSec=5.359682771051617, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1459/5000 [4:55:17<10:49:29, 11.01s/it][2022-12-20 15:17:48,393] [INFO] [timer.py:197:stop] 0/2954, RunningAvgSamplesPerSec=5.8672187644952185, CurrSamplesPerSec=5.302231857858908, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1460/5000 [4:55:28<10:50:14, 11.02s/it][2022-12-20 15:17:59,768] [INFO] [timer.py:197:stop] 0/2956, RunningAvgSamplesPerSec=5.867093372170636, CurrSamplesPerSec=5.0381269772130315, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1461/5000 [4:55:40<10:55:38, 11.12s/it][2022-12-20 15:18:10,739] [INFO] [timer.py:197:stop] 0/2958, RunningAvgSamplesPerSec=5.867114386874484, CurrSamplesPerSec=5.344570261522429, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1462/5000 [4:55:51<10:53:22, 11.08s/it][2022-12-20 15:18:21,779] [INFO] [logging.py:68:log_dist] [Rank 0] step=1480, skipped=3, lr=[7.831111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:18:21,781] [INFO] [timer.py:197:stop] 0/2960, RunningAvgSamplesPerSec=5.867115780039747, CurrSamplesPerSec=5.3189859626643825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1463/5000 [4:56:02<10:52:04, 11.06s/it][2022-12-20 15:18:32,865] [INFO] [timer.py:197:stop] 0/2962, RunningAvgSamplesPerSec=5.867096446134377, CurrSamplesPerSec=5.2485556158247215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1464/5000 [4:56:13<10:52:47, 11.08s/it][2022-12-20 15:18:43,883] [INFO] [timer.py:197:stop] 0/2964, RunningAvgSamplesPerSec=5.867101121467743, CurrSamplesPerSec=5.330315708800027, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1465/5000 [4:56:24<10:50:41, 11.04s/it][2022-12-20 15:18:54,886] [INFO] [timer.py:197:stop] 0/2966, RunningAvgSamplesPerSec=5.86711117318959, CurrSamplesPerSec=5.30045662417831, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1466/5000 [4:56:35<10:50:09, 11.04s/it][2022-12-20 15:19:06,042] [INFO] [timer.py:197:stop] 0/2968, RunningAvgSamplesPerSec=5.867066482677286, CurrSamplesPerSec=5.187811181657828, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1467/5000 [4:56:46<10:52:47, 11.09s/it][2022-12-20 15:19:17,051] [INFO] [timer.py:197:stop] 0/2970, RunningAvgSamplesPerSec=5.867074273623438, CurrSamplesPerSec=5.3516979986734965, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1468/5000 [4:56:57<10:50:55, 11.06s/it][2022-12-20 15:19:28,111] [INFO] [timer.py:197:stop] 0/2972, RunningAvgSamplesPerSec=5.8670641933922445, CurrSamplesPerSec=5.291072368564475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1469/5000 [4:57:08<10:50:01, 11.05s/it][2022-12-20 15:19:39,112] [INFO] [timer.py:197:stop] 0/2974, RunningAvgSamplesPerSec=5.86707476244847, CurrSamplesPerSec=5.3039024334148825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1470/5000 [4:57:19<10:49:19, 11.04s/it][2022-12-20 15:19:50,076] [INFO] [timer.py:197:stop] 0/2976, RunningAvgSamplesPerSec=5.867105391228582, CurrSamplesPerSec=5.351362997924009, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1471/5000 [4:57:30<10:48:22, 11.02s/it][2022-12-20 15:20:01,110] [INFO] [timer.py:197:stop] 0/2978, RunningAvgSamplesPerSec=5.867106639467362, CurrSamplesPerSec=5.318574323737692, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1472/5000 [4:57:41<10:47:37, 11.01s/it][2022-12-20 15:20:12,106] [INFO] [logging.py:68:log_dist] [Rank 0] step=1490, skipped=3, lr=[7.808888888888888e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:20:12,107] [INFO] [timer.py:197:stop] 0/2980, RunningAvgSamplesPerSec=5.867118586350411, CurrSamplesPerSec=5.309225618020818, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1473/5000 [4:57:52<10:48:10, 11.03s/it][2022-12-20 15:20:23,132] [INFO] [timer.py:197:stop] 0/2982, RunningAvgSamplesPerSec=5.867127296532327, CurrSamplesPerSec=5.3374761311717895, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 29%|██▉       | 1474/5000 [4:58:03<10:47:57, 11.03s/it][2022-12-20 15:20:34,183] [INFO] [timer.py:197:stop] 0/2984, RunningAvgSamplesPerSec=5.867120041797028, CurrSamplesPerSec=5.313298641257833, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1475/5000 [4:58:14<10:48:07, 11.03s/it]
                                                        
{'loss': 0.0006, 'learning_rate': 7.804444444444445e-06, 'epoch': 35.96}

 30%|██▉       | 1475/5000 [4:58:14<10:48:07, 11.03s/it][2022-12-20 15:20:45,476] [INFO] [timer.py:197:stop] 0/2986, RunningAvgSamplesPerSec=5.867026159989341, CurrSamplesPerSec=5.104870620413164, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1476/5000 [4:58:26<10:52:18, 11.11s/it][2022-12-20 15:20:55,582] [INFO] [timer.py:197:stop] 0/2988, RunningAvgSamplesPerSec=5.867355876288415, CurrSamplesPerSec=6.267056000842719, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 15:21:06,645] [INFO] [timer.py:197:stop] 0/2990, RunningAvgSamplesPerSec=5.867347890080102, CurrSamplesPerSec=5.301989101229689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1477/5000 [4:58:42<12:22:14, 12.64s/it][2022-12-20 15:21:17,964] [INFO] [timer.py:197:stop] 0/2992, RunningAvgSamplesPerSec=5.867244082505629, CurrSamplesPerSec=5.0847359718844425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1478/5000 [4:58:53<11:58:45, 12.24s/it][2022-12-20 15:21:29,028] [INFO] [timer.py:197:stop] 0/2994, RunningAvgSamplesPerSec=5.8672382996983155, CurrSamplesPerSec=5.303764942579312, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1479/5000 [4:59:04<11:37:46, 11.89s/it][2022-12-20 15:21:40,013] [INFO] [timer.py:197:stop] 0/2996, RunningAvgSamplesPerSec=5.867255114910844, CurrSamplesPerSec=5.3425043498640905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1480/5000 [4:59:15<11:21:38, 11.62s/it][2022-12-20 15:21:51,277] [INFO] [timer.py:197:stop] 0/2998, RunningAvgSamplesPerSec=5.867171447366345, CurrSamplesPerSec=5.101236981591604, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1481/5000 [4:59:26<11:15:12, 11.51s/it][2022-12-20 15:22:02,321] [INFO] [logging.py:68:log_dist] [Rank 0] step=1500, skipped=3, lr=[7.786666666666666e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:22:02,323] [INFO] [timer.py:197:stop] 0/3000, RunningAvgSamplesPerSec=5.867169561864401, CurrSamplesPerSec=5.3275727993868145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1482/5000 [4:59:37<11:06:47, 11.37s/it][2022-12-20 15:22:13,434] [INFO] [timer.py:197:stop] 0/3002, RunningAvgSamplesPerSec=5.867147202143589, CurrSamplesPerSec=5.284803354107506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1483/5000 [4:59:49<11:02:01, 11.29s/it][2022-12-20 15:22:24,477] [INFO] [timer.py:197:stop] 0/3004, RunningAvgSamplesPerSec=5.867143373747431, CurrSamplesPerSec=5.289350456217155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1484/5000 [5:00:00<10:57:25, 11.22s/it][2022-12-20 15:22:35,515] [INFO] [timer.py:197:stop] 0/3006, RunningAvgSamplesPerSec=5.867147431483427, CurrSamplesPerSec=5.353116989083556, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1485/5000 [5:00:11<10:54:02, 11.16s/it][2022-12-20 15:22:46,572] [INFO] [timer.py:197:stop] 0/3008, RunningAvgSamplesPerSec=5.867144781786277, CurrSamplesPerSec=5.310961758172568, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1486/5000 [5:00:22<10:51:58, 11.13s/it][2022-12-20 15:22:57,785] [INFO] [timer.py:197:stop] 0/3010, RunningAvgSamplesPerSec=5.867079697566178, CurrSamplesPerSec=5.172297534766197, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1487/5000 [5:00:33<10:53:12, 11.16s/it][2022-12-20 15:23:08,780] [INFO] [timer.py:197:stop] 0/3012, RunningAvgSamplesPerSec=5.867096655425602, CurrSamplesPerSec=5.328094970156685, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1488/5000 [5:00:44<10:50:11, 11.11s/it][2022-12-20 15:23:19,827] [INFO] [timer.py:197:stop] 0/3014, RunningAvgSamplesPerSec=5.867090955722156, CurrSamplesPerSec=5.312953498361217, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1489/5000 [5:00:55<10:48:56, 11.09s/it][2022-12-20 15:23:30,908] [INFO] [timer.py:197:stop] 0/3016, RunningAvgSamplesPerSec=5.867073578279416, CurrSamplesPerSec=5.282343438659136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1490/5000 [5:01:06<10:48:35, 11.09s/it][2022-12-20 15:23:41,964] [INFO] [timer.py:197:stop] 0/3018, RunningAvgSamplesPerSec=5.867066200345507, CurrSamplesPerSec=5.31085647339549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1491/5000 [5:01:17<10:47:52, 11.08s/it][2022-12-20 15:23:53,019] [INFO] [logging.py:68:log_dist] [Rank 0] step=1510, skipped=3, lr=[7.764444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:23:53,020] [INFO] [timer.py:197:stop] 0/3020, RunningAvgSamplesPerSec=5.867057125320396, CurrSamplesPerSec=5.2829977642724435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1492/5000 [5:01:28<10:47:18, 11.07s/it][2022-12-20 15:24:04,062] [INFO] [timer.py:197:stop] 0/3022, RunningAvgSamplesPerSec=5.86705310962787, CurrSamplesPerSec=5.32072132157189, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1493/5000 [5:01:39<10:46:35, 11.06s/it][2022-12-20 15:24:15,169] [INFO] [timer.py:197:stop] 0/3024, RunningAvgSamplesPerSec=5.867045179116094, CurrSamplesPerSec=5.322561447735841, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1494/5000 [5:01:50<10:47:11, 11.08s/it][2022-12-20 15:24:26,117] [INFO] [timer.py:197:stop] 0/3026, RunningAvgSamplesPerSec=5.867074426057374, CurrSamplesPerSec=5.3719449647689785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1495/5000 [5:02:01<10:44:46, 11.04s/it][2022-12-20 15:24:37,192] [INFO] [timer.py:197:stop] 0/3028, RunningAvgSamplesPerSec=5.867058263240498, CurrSamplesPerSec=5.277408268189673, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1496/5000 [5:02:12<10:45:14, 11.05s/it][2022-12-20 15:24:48,178] [INFO] [timer.py:197:stop] 0/3030, RunningAvgSamplesPerSec=5.867074493580833, CurrSamplesPerSec=5.344055708443579, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1497/5000 [5:02:23<10:43:57, 11.03s/it][2022-12-20 15:24:59,168] [INFO] [timer.py:197:stop] 0/3032, RunningAvgSamplesPerSec=5.8670892522203175, CurrSamplesPerSec=5.362276533070928, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1498/5000 [5:02:34<10:43:05, 11.02s/it][2022-12-20 15:25:10,224] [INFO] [timer.py:197:stop] 0/3034, RunningAvgSamplesPerSec=5.8670807024635305, CurrSamplesPerSec=5.333892880964507, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|██▉       | 1499/5000 [5:02:45<10:43:34, 11.03s/it][2022-12-20 15:25:21,252] [INFO] [timer.py:197:stop] 0/3036, RunningAvgSamplesPerSec=5.86708176744793, CurrSamplesPerSec=5.327406800556645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1500/5000 [5:02:56<10:43:21, 11.03s/it]
                                                        
{'loss': 0.0005, 'learning_rate': 7.746666666666666e-06, 'epoch': 36.58}

 30%|███       | 1500/5000 [5:02:56<10:43:21, 11.03s/it][2022-12-20 15:25:32,239] [INFO] [timer.py:197:stop] 0/3038, RunningAvgSamplesPerSec=5.86709766334428, CurrSamplesPerSec=5.338563108297541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1501/5000 [5:03:07<10:42:26, 11.02s/it][2022-12-20 15:25:43,260] [INFO] [logging.py:68:log_dist] [Rank 0] step=1520, skipped=3, lr=[7.742222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:25:43,262] [INFO] [timer.py:197:stop] 0/3040, RunningAvgSamplesPerSec=5.867100782471541, CurrSamplesPerSec=5.32822166857146, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1502/5000 [5:03:18<10:42:21, 11.02s/it][2022-12-20 15:25:54,379] [INFO] [timer.py:197:stop] 0/3042, RunningAvgSamplesPerSec=5.86707693277007, CurrSamplesPerSec=5.345282457479255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1503/5000 [5:03:29<10:43:54, 11.05s/it][2022-12-20 15:26:05,460] [INFO] [timer.py:197:stop] 0/3044, RunningAvgSamplesPerSec=5.867059272516083, CurrSamplesPerSec=5.2841802029652465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1504/5000 [5:03:41<10:44:18, 11.06s/it][2022-12-20 15:26:16,483] [INFO] [timer.py:197:stop] 0/3046, RunningAvgSamplesPerSec=5.867061892689114, CurrSamplesPerSec=5.3409679189900485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1505/5000 [5:03:52<10:43:31, 11.05s/it][2022-12-20 15:26:27,517] [INFO] [timer.py:197:stop] 0/3048, RunningAvgSamplesPerSec=5.867061267464411, CurrSamplesPerSec=5.299335309490053, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1506/5000 [5:04:03<10:43:06, 11.04s/it][2022-12-20 15:26:38,506] [INFO] [timer.py:197:stop] 0/3050, RunningAvgSamplesPerSec=5.8670764339901655, CurrSamplesPerSec=5.320373315612726, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1507/5000 [5:04:14<10:41:57, 11.03s/it][2022-12-20 15:26:49,467] [INFO] [timer.py:197:stop] 0/3052, RunningAvgSamplesPerSec=5.867101363899046, CurrSamplesPerSec=5.347604691134345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1508/5000 [5:04:25<10:40:37, 11.01s/it][2022-12-20 15:27:00,560] [INFO] [timer.py:197:stop] 0/3054, RunningAvgSamplesPerSec=5.867085886600037, CurrSamplesPerSec=5.2962941610466086, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1509/5000 [5:04:36<10:41:55, 11.03s/it][2022-12-20 15:27:11,600] [INFO] [timer.py:197:stop] 0/3056, RunningAvgSamplesPerSec=5.867082602322803, CurrSamplesPerSec=5.326687098547047, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1510/5000 [5:04:47<10:41:52, 11.04s/it][2022-12-20 15:27:22,628] [INFO] [timer.py:197:stop] 0/3058, RunningAvgSamplesPerSec=5.867083961108254, CurrSamplesPerSec=5.328651938211841, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1511/5000 [5:04:58<10:41:33, 11.03s/it][2022-12-20 15:27:33,693] [INFO] [logging.py:68:log_dist] [Rank 0] step=1530, skipped=3, lr=[7.72e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:27:33,695] [INFO] [timer.py:197:stop] 0/3060, RunningAvgSamplesPerSec=5.867077621120877, CurrSamplesPerSec=5.309095411265069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1512/5000 [5:05:09<10:41:58, 11.04s/it][2022-12-20 15:27:44,749] [INFO] [timer.py:197:stop] 0/3062, RunningAvgSamplesPerSec=5.867069595993426, CurrSamplesPerSec=5.297249436985049, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1513/5000 [5:05:20<10:41:58, 11.05s/it][2022-12-20 15:27:55,721] [INFO] [timer.py:197:stop] 0/3064, RunningAvgSamplesPerSec=5.86709092813864, CurrSamplesPerSec=5.357057733711487, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1514/5000 [5:05:31<10:40:29, 11.02s/it][2022-12-20 15:28:06,843] [INFO] [timer.py:197:stop] 0/3066, RunningAvgSamplesPerSec=5.86705909310525, CurrSamplesPerSec=5.324130600413189, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1515/5000 [5:05:42<10:42:01, 11.05s/it][2022-12-20 15:28:17,862] [INFO] [timer.py:197:stop] 0/3068, RunningAvgSamplesPerSec=5.8670641199060265, CurrSamplesPerSec=5.31152608006618, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1516/5000 [5:05:53<10:41:13, 11.04s/it][2022-12-20 15:28:28,829] [INFO] [timer.py:197:stop] 0/3070, RunningAvgSamplesPerSec=5.867086311420677, CurrSamplesPerSec=5.3414888926032935, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1517/5000 [5:06:04<10:39:44, 11.02s/it][2022-12-20 15:28:39,013] [INFO] [timer.py:197:stop] 0/3072, RunningAvgSamplesPerSec=5.867379823670472, CurrSamplesPerSec=5.3178323542999415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1518/5000 [5:06:19<11:51:06, 12.25s/it][2022-12-20 15:28:50,007] [INFO] [timer.py:197:stop] 0/3074, RunningAvgSamplesPerSec=5.867392451654223, CurrSamplesPerSec=5.347825220773852, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1519/5000 [5:06:30<11:29:22, 11.88s/it][2022-12-20 15:29:01,059] [INFO] [timer.py:197:stop] 0/3076, RunningAvgSamplesPerSec=5.8673846384449435, CurrSamplesPerSec=5.316255540434527, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1520/5000 [5:06:41<11:14:07, 11.62s/it][2022-12-20 15:29:12,152] [INFO] [timer.py:197:stop] 0/3078, RunningAvgSamplesPerSec=5.867362195003146, CurrSamplesPerSec=5.249550620378161, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1521/5000 [5:06:52<11:04:32, 11.46s/it][2022-12-20 15:29:23,130] [INFO] [logging.py:68:log_dist] [Rank 0] step=1540, skipped=3, lr=[7.697777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:29:23,131] [INFO] [timer.py:197:stop] 0/3080, RunningAvgSamplesPerSec=5.867379931724516, CurrSamplesPerSec=5.3407003510788025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1522/5000 [5:07:03<10:56:27, 11.32s/it][2022-12-20 15:29:34,192] [INFO] [timer.py:197:stop] 0/3082, RunningAvgSamplesPerSec=5.867369149413616, CurrSamplesPerSec=5.293720185590196, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1523/5000 [5:07:14<10:52:15, 11.26s/it][2022-12-20 15:29:45,290] [INFO] [timer.py:197:stop] 0/3084, RunningAvgSamplesPerSec=5.867347823001926, CurrSamplesPerSec=5.29513303467783, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1524/5000 [5:07:25<10:48:20, 11.19s/it][2022-12-20 15:29:56,259] [INFO] [timer.py:197:stop] 0/3086, RunningAvgSamplesPerSec=5.867368933240322, CurrSamplesPerSec=5.353656778601285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 30%|███       | 1525/5000 [5:07:36<10:44:30, 11.13s/it]
                                                        
{'loss': 0.0004, 'learning_rate': 7.691111111111112e-06, 'epoch': 37.19}

 30%|███       | 1525/5000 [5:07:36<10:44:30, 11.13s/it][2022-12-20 15:30:07,287] [INFO] [timer.py:197:stop] 0/3088, RunningAvgSamplesPerSec=5.867374850148722, CurrSamplesPerSec=5.314342335044032, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1526/5000 [5:07:47<10:42:46, 11.10s/it][2022-12-20 15:30:18,342] [INFO] [timer.py:197:stop] 0/3090, RunningAvgSamplesPerSec=5.867366324211938, CurrSamplesPerSec=5.29861605953942, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1527/5000 [5:07:58<10:41:40, 11.09s/it][2022-12-20 15:30:29,364] [INFO] [timer.py:197:stop] 0/3092, RunningAvgSamplesPerSec=5.867369688964089, CurrSamplesPerSec=5.3219747310963506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1528/5000 [5:08:09<10:40:43, 11.07s/it][2022-12-20 15:30:40,371] [INFO] [timer.py:197:stop] 0/3094, RunningAvgSamplesPerSec=5.8673812788436726, CurrSamplesPerSec=5.353147306653666, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1529/5000 [5:08:20<10:38:42, 11.04s/it][2022-12-20 15:30:51,422] [INFO] [timer.py:197:stop] 0/3096, RunningAvgSamplesPerSec=5.867374205248553, CurrSamplesPerSec=5.27852841795515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1530/5000 [5:08:31<10:39:14, 11.05s/it][2022-12-20 15:31:02,415] [INFO] [timer.py:197:stop] 0/3098, RunningAvgSamplesPerSec=5.867387605115355, CurrSamplesPerSec=5.356932225917254, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1531/5000 [5:08:42<10:38:01, 11.04s/it][2022-12-20 15:31:13,446] [INFO] [logging.py:68:log_dist] [Rank 0] step=1550, skipped=3, lr=[7.675555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:31:13,448] [INFO] [timer.py:197:stop] 0/3100, RunningAvgSamplesPerSec=5.867387245830778, CurrSamplesPerSec=5.3225557487866615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1532/5000 [5:08:53<10:37:13, 11.02s/it][2022-12-20 15:31:24,606] [INFO] [timer.py:197:stop] 0/3102, RunningAvgSamplesPerSec=5.867342952325706, CurrSamplesPerSec=5.185365986521761, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1533/5000 [5:09:05<10:39:51, 11.07s/it][2022-12-20 15:31:35,638] [INFO] [timer.py:197:stop] 0/3104, RunningAvgSamplesPerSec=5.867342723093399, CurrSamplesPerSec=5.320308570436001, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1534/5000 [5:09:16<10:38:19, 11.05s/it][2022-12-20 15:31:46,590] [INFO] [timer.py:197:stop] 0/3106, RunningAvgSamplesPerSec=5.867369856408551, CurrSamplesPerSec=5.358584182359116, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1535/5000 [5:09:27<10:36:53, 11.03s/it][2022-12-20 15:31:58,001] [INFO] [timer.py:197:stop] 0/3108, RunningAvgSamplesPerSec=5.867238432312703, CurrSamplesPerSec=4.996465448998019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1536/5000 [5:09:38<10:42:37, 11.13s/it][2022-12-20 15:32:08,955] [INFO] [timer.py:197:stop] 0/3110, RunningAvgSamplesPerSec=5.867264888302843, CurrSamplesPerSec=5.342848876581525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1537/5000 [5:09:49<10:40:55, 11.10s/it][2022-12-20 15:32:20,029] [INFO] [timer.py:197:stop] 0/3112, RunningAvgSamplesPerSec=5.867250022897016, CurrSamplesPerSec=5.315813585337739, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1538/5000 [5:10:00<10:39:28, 11.08s/it][2022-12-20 15:32:31,214] [INFO] [timer.py:197:stop] 0/3114, RunningAvgSamplesPerSec=5.86719718814154, CurrSamplesPerSec=5.183987472320314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1539/5000 [5:10:11<10:40:40, 11.11s/it][2022-12-20 15:32:42,178] [INFO] [timer.py:197:stop] 0/3116, RunningAvgSamplesPerSec=5.867220329416286, CurrSamplesPerSec=5.356111331902911, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1540/5000 [5:10:22<10:37:06, 11.05s/it][2022-12-20 15:32:53,132] [INFO] [timer.py:197:stop] 0/3118, RunningAvgSamplesPerSec=5.867246327999638, CurrSamplesPerSec=5.317448069815289, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1541/5000 [5:10:33<10:36:09, 11.03s/it][2022-12-20 15:33:04,488] [INFO] [logging.py:68:log_dist] [Rank 0] step=1560, skipped=3, lr=[7.653333333333333e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:33:04,490] [INFO] [timer.py:197:stop] 0/3120, RunningAvgSamplesPerSec=5.8671330803190855, CurrSamplesPerSec=5.022460492732595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1542/5000 [5:10:45<10:42:07, 11.14s/it][2022-12-20 15:33:15,521] [INFO] [timer.py:197:stop] 0/3122, RunningAvgSamplesPerSec=5.8671325232473075, CurrSamplesPerSec=5.322616327130317, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1543/5000 [5:10:56<10:39:46, 11.10s/it][2022-12-20 15:33:26,543] [INFO] [timer.py:197:stop] 0/3124, RunningAvgSamplesPerSec=5.867135794683141, CurrSamplesPerSec=5.317427424523699, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1544/5000 [5:11:07<10:40:53, 11.13s/it][2022-12-20 15:33:37,983] [INFO] [timer.py:197:stop] 0/3126, RunningAvgSamplesPerSec=5.867089641406597, CurrSamplesPerSec=5.320040959697411, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1545/5000 [5:11:18<10:42:46, 11.16s/it][2022-12-20 15:33:48,980] [INFO] [timer.py:197:stop] 0/3128, RunningAvgSamplesPerSec=5.867101211428546, CurrSamplesPerSec=5.307833996520625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1546/5000 [5:11:29<10:40:28, 11.13s/it][2022-12-20 15:34:00,027] [INFO] [timer.py:197:stop] 0/3130, RunningAvgSamplesPerSec=5.867095764038956, CurrSamplesPerSec=5.3021901750277785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1547/5000 [5:11:40<10:39:10, 11.11s/it][2022-12-20 15:34:11,443] [INFO] [timer.py:197:stop] 0/3132, RunningAvgSamplesPerSec=5.86696327746274, CurrSamplesPerSec=5.008381138957689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1548/5000 [5:11:51<10:44:00, 11.19s/it][2022-12-20 15:34:22,446] [INFO] [timer.py:197:stop] 0/3134, RunningAvgSamplesPerSec=5.866972619652037, CurrSamplesPerSec=5.336698110458002, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1549/5000 [5:12:03<10:41:05, 11.15s/it][2022-12-20 15:34:33,530] [INFO] [timer.py:197:stop] 0/3136, RunningAvgSamplesPerSec=5.866954316963651, CurrSamplesPerSec=5.2938504744726345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1550/5000 [5:12:14<10:39:18, 11.12s/it]
                                                        
{'loss': 0.0004, 'learning_rate': 7.635555555555556e-06, 'epoch': 37.8}

 31%|███       | 1550/5000 [5:12:14<10:39:18, 11.12s/it][2022-12-20 15:34:44,898] [INFO] [timer.py:197:stop] 0/3138, RunningAvgSamplesPerSec=5.866839421662756, CurrSamplesPerSec=5.033475428352372, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1551/5000 [5:12:25<10:42:50, 11.18s/it][2022-12-20 15:34:55,864] [INFO] [logging.py:68:log_dist] [Rank 0] step=1570, skipped=3, lr=[7.631111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:34:55,866] [INFO] [timer.py:197:stop] 0/3140, RunningAvgSamplesPerSec=5.866861278404587, CurrSamplesPerSec=5.338900118633625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1552/5000 [5:12:36<10:40:15, 11.14s/it][2022-12-20 15:35:06,899] [INFO] [timer.py:197:stop] 0/3142, RunningAvgSamplesPerSec=5.86686099948287, CurrSamplesPerSec=5.349256228768708, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1553/5000 [5:12:47<10:37:47, 11.10s/it][2022-12-20 15:35:17,967] [INFO] [timer.py:197:stop] 0/3144, RunningAvgSamplesPerSec=5.866848782413336, CurrSamplesPerSec=5.297122325696289, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1554/5000 [5:12:58<10:37:06, 11.09s/it][2022-12-20 15:35:29,003] [INFO] [timer.py:197:stop] 0/3146, RunningAvgSamplesPerSec=5.866847256798957, CurrSamplesPerSec=5.329595009044819, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1555/5000 [5:13:09<10:34:54, 11.06s/it][2022-12-20 15:35:40,015] [INFO] [timer.py:197:stop] 0/3148, RunningAvgSamplesPerSec=5.866853990259572, CurrSamplesPerSec=5.2978100119962575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1556/5000 [5:13:20<10:35:10, 11.07s/it][2022-12-20 15:35:51,153] [INFO] [timer.py:197:stop] 0/3150, RunningAvgSamplesPerSec=5.8668173622743085, CurrSamplesPerSec=5.250824743970036, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1557/5000 [5:13:31<10:35:24, 11.07s/it][2022-12-20 15:36:02,190] [INFO] [timer.py:197:stop] 0/3152, RunningAvgSamplesPerSec=5.866815552610913, CurrSamplesPerSec=5.297470223723035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1558/5000 [5:13:42<10:34:15, 11.06s/it][2022-12-20 15:36:12,277] [INFO] [timer.py:197:stop] 0/3154, RunningAvgSamplesPerSec=5.86713429495683, CurrSamplesPerSec=6.24804235631808, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 15:36:23,356] [INFO] [timer.py:197:stop] 0/3156, RunningAvgSamplesPerSec=5.867117181613105, CurrSamplesPerSec=5.25697197048839, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1559/5000 [5:13:58<12:03:29, 12.62s/it][2022-12-20 15:36:34,378] [INFO] [timer.py:197:stop] 0/3158, RunningAvgSamplesPerSec=5.867120232068444, CurrSamplesPerSec=5.307669855132011, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1560/5000 [5:14:09<11:35:53, 12.14s/it][2022-12-20 15:36:45,366] [INFO] [logging.py:68:log_dist] [Rank 0] step=1580, skipped=3, lr=[7.608888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:36:45,367] [INFO] [timer.py:197:stop] 0/3160, RunningAvgSamplesPerSec=5.867134621129926, CurrSamplesPerSec=5.361038331913165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1561/5000 [5:14:20<11:15:56, 11.79s/it][2022-12-20 15:36:56,393] [INFO] [timer.py:197:stop] 0/3162, RunningAvgSamplesPerSec=5.867136629268005, CurrSamplesPerSec=5.310088924328765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███       | 1562/5000 [5:14:31<11:02:33, 11.56s/it][2022-12-20 15:37:07,454] [INFO] [timer.py:197:stop] 0/3164, RunningAvgSamplesPerSec=5.867126930946481, CurrSamplesPerSec=5.301556215972514, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1563/5000 [5:14:43<10:53:44, 11.41s/it][2022-12-20 15:37:18,431] [INFO] [timer.py:197:stop] 0/3166, RunningAvgSamplesPerSec=5.8671455042925995, CurrSamplesPerSec=5.332084736214061, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1564/5000 [5:14:54<10:46:03, 11.28s/it][2022-12-20 15:37:29,469] [INFO] [timer.py:197:stop] 0/3168, RunningAvgSamplesPerSec=5.867143193945456, CurrSamplesPerSec=5.3121220584991615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1565/5000 [5:15:05<10:41:41, 11.21s/it][2022-12-20 15:37:40,476] [INFO] [timer.py:197:stop] 0/3170, RunningAvgSamplesPerSec=5.867151760285315, CurrSamplesPerSec=5.347842480382645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1566/5000 [5:15:16<10:38:02, 11.15s/it][2022-12-20 15:37:51,524] [INFO] [timer.py:197:stop] 0/3172, RunningAvgSamplesPerSec=5.867146126950085, CurrSamplesPerSec=5.297617216008058, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1567/5000 [5:15:27<10:36:08, 11.12s/it][2022-12-20 15:38:02,476] [INFO] [timer.py:197:stop] 0/3174, RunningAvgSamplesPerSec=5.867173263021302, CurrSamplesPerSec=5.351576796108012, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1568/5000 [5:15:38<10:33:05, 11.07s/it][2022-12-20 15:38:13,454] [INFO] [timer.py:197:stop] 0/3176, RunningAvgSamplesPerSec=5.867191318574457, CurrSamplesPerSec=5.328544046966301, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1569/5000 [5:15:49<10:31:21, 11.04s/it][2022-12-20 15:38:24,456] [INFO] [timer.py:197:stop] 0/3178, RunningAvgSamplesPerSec=5.8672017471547875, CurrSamplesPerSec=5.342220680356176, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1570/5000 [5:16:00<10:30:31, 11.03s/it][2022-12-20 15:38:35,435] [INFO] [logging.py:68:log_dist] [Rank 0] step=1590, skipped=3, lr=[7.586666666666668e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:38:35,437] [INFO] [timer.py:197:stop] 0/3180, RunningAvgSamplesPerSec=5.867219361530512, CurrSamplesPerSec=5.301566686479741, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1571/5000 [5:16:11<10:29:29, 11.01s/it][2022-12-20 15:38:46,465] [INFO] [timer.py:197:stop] 0/3182, RunningAvgSamplesPerSec=5.8672202113364555, CurrSamplesPerSec=5.333258735852439, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1572/5000 [5:16:22<10:29:32, 11.02s/it][2022-12-20 15:38:57,465] [INFO] [timer.py:197:stop] 0/3184, RunningAvgSamplesPerSec=5.867231058896733, CurrSamplesPerSec=5.351747718919474, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1573/5000 [5:16:33<10:29:01, 11.01s/it][2022-12-20 15:39:08,700] [INFO] [timer.py:197:stop] 0/3186, RunningAvgSamplesPerSec=5.86716228901257, CurrSamplesPerSec=5.154021646873344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 31%|███▏      | 1574/5000 [5:16:44<10:32:39, 11.08s/it][2022-12-20 15:39:19,708] [INFO] [timer.py:197:stop] 0/3188, RunningAvgSamplesPerSec=5.867170208009518, CurrSamplesPerSec=5.340776431998533, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1575/5000 [5:16:55<10:31:14, 11.06s/it]
                                                        
{'loss': 0.0004, 'learning_rate': 7.5777777777777785e-06, 'epoch': 38.41}

 32%|███▏      | 1575/5000 [5:16:55<10:31:14, 11.06s/it][2022-12-20 15:39:30,710] [INFO] [timer.py:197:stop] 0/3190, RunningAvgSamplesPerSec=5.867180775106877, CurrSamplesPerSec=5.326864680265231, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1576/5000 [5:17:06<10:30:05, 11.04s/it][2022-12-20 15:39:41,797] [INFO] [timer.py:197:stop] 0/3192, RunningAvgSamplesPerSec=5.86716179843869, CurrSamplesPerSec=5.296212445560194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1577/5000 [5:17:17<10:30:41, 11.06s/it][2022-12-20 15:39:52,752] [INFO] [timer.py:197:stop] 0/3194, RunningAvgSamplesPerSec=5.867187393996425, CurrSamplesPerSec=5.374365317034259, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1578/5000 [5:17:28<10:28:47, 11.03s/it][2022-12-20 15:40:03,775] [INFO] [timer.py:197:stop] 0/3196, RunningAvgSamplesPerSec=5.8671907073075715, CurrSamplesPerSec=5.30478078240873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1579/5000 [5:17:39<10:28:34, 11.02s/it][2022-12-20 15:40:15,050] [INFO] [timer.py:197:stop] 0/3198, RunningAvgSamplesPerSec=5.867108892358042, CurrSamplesPerSec=5.105812858253662, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1580/5000 [5:17:50<10:32:40, 11.10s/it][2022-12-20 15:40:26,024] [INFO] [logging.py:68:log_dist] [Rank 0] step=1600, skipped=3, lr=[7.564444444444446e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:40:26,025] [INFO] [timer.py:197:stop] 0/3200, RunningAvgSamplesPerSec=5.867127992926814, CurrSamplesPerSec=5.364184537626203, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1581/5000 [5:18:01<10:30:21, 11.06s/it][2022-12-20 15:40:37,000] [INFO] [timer.py:197:stop] 0/3202, RunningAvgSamplesPerSec=5.8671467671042, CurrSamplesPerSec=5.350589242518427, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1582/5000 [5:18:12<10:28:41, 11.04s/it][2022-12-20 15:40:48,082] [INFO] [timer.py:197:stop] 0/3204, RunningAvgSamplesPerSec=5.867130520231536, CurrSamplesPerSec=5.282294791750996, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1583/5000 [5:18:23<10:29:16, 11.05s/it][2022-12-20 15:40:59,129] [INFO] [timer.py:197:stop] 0/3206, RunningAvgSamplesPerSec=5.867130480542053, CurrSamplesPerSec=5.326724305164637, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1584/5000 [5:18:34<10:29:03, 11.05s/it][2022-12-20 15:41:10,137] [INFO] [timer.py:197:stop] 0/3208, RunningAvgSamplesPerSec=5.86714413135889, CurrSamplesPerSec=5.347153887136331, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1585/5000 [5:18:45<10:28:10, 11.04s/it][2022-12-20 15:41:21,452] [INFO] [timer.py:197:stop] 0/3210, RunningAvgSamplesPerSec=5.8670514666614375, CurrSamplesPerSec=5.084840765489478, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1586/5000 [5:18:57<10:33:02, 11.13s/it][2022-12-20 15:41:32,564] [INFO] [timer.py:197:stop] 0/3212, RunningAvgSamplesPerSec=5.867064017678457, CurrSamplesPerSec=5.312880942030667, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1587/5000 [5:19:08<10:32:19, 11.12s/it][2022-12-20 15:41:43,598] [INFO] [timer.py:197:stop] 0/3214, RunningAvgSamplesPerSec=5.867068841035827, CurrSamplesPerSec=5.304525842572113, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1588/5000 [5:19:19<10:30:44, 11.09s/it][2022-12-20 15:41:54,773] [INFO] [timer.py:197:stop] 0/3216, RunningAvgSamplesPerSec=5.86702143610355, CurrSamplesPerSec=5.152992685137411, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1589/5000 [5:19:30<10:32:10, 11.12s/it][2022-12-20 15:42:05,891] [INFO] [timer.py:197:stop] 0/3218, RunningAvgSamplesPerSec=5.867028760220661, CurrSamplesPerSec=5.319026645239607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1590/5000 [5:19:41<10:31:45, 11.12s/it][2022-12-20 15:42:16,922] [INFO] [logging.py:68:log_dist] [Rank 0] step=1610, skipped=3, lr=[7.542222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:42:16,924] [INFO] [timer.py:197:stop] 0/3220, RunningAvgSamplesPerSec=5.867028295885633, CurrSamplesPerSec=5.311424766522087, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1591/5000 [5:19:52<10:30:09, 11.09s/it][2022-12-20 15:42:28,112] [INFO] [timer.py:197:stop] 0/3222, RunningAvgSamplesPerSec=5.866980144467691, CurrSamplesPerSec=5.153907055611606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1592/5000 [5:20:03<10:31:52, 11.12s/it][2022-12-20 15:42:39,277] [INFO] [timer.py:197:stop] 0/3224, RunningAvgSamplesPerSec=5.866989059831445, CurrSamplesPerSec=5.2976627999099435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1593/5000 [5:20:14<10:32:07, 11.13s/it][2022-12-20 15:42:50,346] [INFO] [timer.py:197:stop] 0/3226, RunningAvgSamplesPerSec=5.866982210521485, CurrSamplesPerSec=5.3007815131898175, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1594/5000 [5:20:25<10:30:51, 11.11s/it][2022-12-20 15:43:01,407] [INFO] [timer.py:197:stop] 0/3228, RunningAvgSamplesPerSec=5.8669723317190465, CurrSamplesPerSec=5.268118402867236, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1595/5000 [5:20:37<10:29:47, 11.10s/it][2022-12-20 15:43:12,752] [INFO] [timer.py:197:stop] 0/3230, RunningAvgSamplesPerSec=5.866945840496718, CurrSamplesPerSec=5.28435038458257, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1596/5000 [5:20:48<10:33:49, 11.17s/it][2022-12-20 15:43:23,814] [INFO] [timer.py:197:stop] 0/3232, RunningAvgSamplesPerSec=5.866935915107837, CurrSamplesPerSec=5.305842736223007, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1597/5000 [5:20:59<10:31:45, 11.14s/it][2022-12-20 15:43:34,849] [INFO] [timer.py:197:stop] 0/3234, RunningAvgSamplesPerSec=5.86693479579333, CurrSamplesPerSec=5.278347816918693, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1598/5000 [5:21:10<10:29:48, 11.11s/it][2022-12-20 15:43:45,857] [INFO] [timer.py:197:stop] 0/3236, RunningAvgSamplesPerSec=5.866943393858082, CurrSamplesPerSec=5.335462784298479, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1599/5000 [5:21:21<10:27:55, 11.08s/it][2022-12-20 15:43:55,988] [INFO] [timer.py:197:stop] 0/3238, RunningAvgSamplesPerSec=5.867239463671098, CurrSamplesPerSec=5.295807458195084, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1600/5000 [5:21:36<11:34:58, 12.26s/it]
                                                        
{'loss': 0.0003, 'learning_rate': 7.5222222222222226e-06, 'epoch': 39.02}

 32%|███▏      | 1600/5000 [5:21:36<11:34:58, 12.26s/it][2022-12-20 15:44:07,093] [INFO] [logging.py:68:log_dist] [Rank 0] step=1620, skipped=3, lr=[7.520000000000001e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:44:07,094] [INFO] [timer.py:197:stop] 0/3240, RunningAvgSamplesPerSec=5.867214607599297, CurrSamplesPerSec=5.212364659041231, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1601/5000 [5:21:47<11:16:38, 11.94s/it][2022-12-20 15:44:18,143] [INFO] [timer.py:197:stop] 0/3242, RunningAvgSamplesPerSec=5.867217130881242, CurrSamplesPerSec=5.3400947562580665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1602/5000 [5:21:58<10:59:51, 11.65s/it][2022-12-20 15:44:29,170] [INFO] [timer.py:197:stop] 0/3244, RunningAvgSamplesPerSec=5.867218672121656, CurrSamplesPerSec=5.288177787275803, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1603/5000 [5:22:09<10:49:38, 11.47s/it][2022-12-20 15:44:40,366] [INFO] [timer.py:197:stop] 0/3246, RunningAvgSamplesPerSec=5.867165421614602, CurrSamplesPerSec=5.177446780408352, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1604/5000 [5:22:21<10:49:42, 11.48s/it][2022-12-20 15:44:51,705] [INFO] [timer.py:197:stop] 0/3248, RunningAvgSamplesPerSec=5.867153315118779, CurrSamplesPerSec=5.3017388275477355, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1605/5000 [5:22:32<10:42:11, 11.35s/it][2022-12-20 15:45:02,750] [INFO] [timer.py:197:stop] 0/3250, RunningAvgSamplesPerSec=5.867148549917278, CurrSamplesPerSec=5.303427742952233, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1606/5000 [5:22:43<10:36:48, 11.26s/it][2022-12-20 15:45:14,057] [INFO] [timer.py:197:stop] 0/3252, RunningAvgSamplesPerSec=5.867059051494024, CurrSamplesPerSec=5.0872456552056535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1607/5000 [5:22:54<10:38:22, 11.29s/it][2022-12-20 15:45:25,104] [INFO] [timer.py:197:stop] 0/3254, RunningAvgSamplesPerSec=5.86707599163612, CurrSamplesPerSec=5.34845196515103, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1608/5000 [5:23:05<10:32:54, 11.20s/it][2022-12-20 15:45:36,104] [INFO] [timer.py:197:stop] 0/3256, RunningAvgSamplesPerSec=5.86708635248154, CurrSamplesPerSec=5.329766858661181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1609/5000 [5:23:16<10:28:59, 11.13s/it][2022-12-20 15:45:47,465] [INFO] [timer.py:197:stop] 0/3258, RunningAvgSamplesPerSec=5.866977081469949, CurrSamplesPerSec=5.007083340996541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1610/5000 [5:23:27<10:33:02, 11.20s/it][2022-12-20 15:45:58,417] [INFO] [logging.py:68:log_dist] [Rank 0] step=1630, skipped=3, lr=[7.4977777777777785e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:45:58,419] [INFO] [timer.py:197:stop] 0/3260, RunningAvgSamplesPerSec=5.867002709649023, CurrSamplesPerSec=5.363881412173787, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1611/5000 [5:23:38<10:29:02, 11.14s/it][2022-12-20 15:46:09,454] [INFO] [timer.py:197:stop] 0/3262, RunningAvgSamplesPerSec=5.867001103606801, CurrSamplesPerSec=5.314345912204189, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1612/5000 [5:23:50<10:27:24, 11.11s/it][2022-12-20 15:46:20,459] [INFO] [timer.py:197:stop] 0/3264, RunningAvgSamplesPerSec=5.867010073049012, CurrSamplesPerSec=5.356330853650448, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1613/5000 [5:24:01<10:26:06, 11.09s/it][2022-12-20 15:46:31,534] [INFO] [timer.py:197:stop] 0/3266, RunningAvgSamplesPerSec=5.866995595406668, CurrSamplesPerSec=5.328680075274538, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1614/5000 [5:24:12<10:24:59, 11.07s/it][2022-12-20 15:46:42,595] [INFO] [timer.py:197:stop] 0/3268, RunningAvgSamplesPerSec=5.866985727428108, CurrSamplesPerSec=5.307075714612886, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1615/5000 [5:24:23<10:24:07, 11.06s/it][2022-12-20 15:46:53,611] [INFO] [timer.py:197:stop] 0/3270, RunningAvgSamplesPerSec=5.866990808636021, CurrSamplesPerSec=5.323398903762646, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1616/5000 [5:24:34<10:23:43, 11.06s/it][2022-12-20 15:47:04,619] [INFO] [timer.py:197:stop] 0/3272, RunningAvgSamplesPerSec=5.866998901489133, CurrSamplesPerSec=5.361418234571171, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1617/5000 [5:24:45<10:22:21, 11.04s/it][2022-12-20 15:47:15,673] [INFO] [timer.py:197:stop] 0/3274, RunningAvgSamplesPerSec=5.86699155327955, CurrSamplesPerSec=5.3029602611430136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1618/5000 [5:24:56<10:22:22, 11.04s/it][2022-12-20 15:47:26,676] [INFO] [timer.py:197:stop] 0/3276, RunningAvgSamplesPerSec=5.8670009866415755, CurrSamplesPerSec=5.344717538126158, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1619/5000 [5:25:07<10:21:26, 11.03s/it][2022-12-20 15:47:37,698] [INFO] [timer.py:197:stop] 0/3278, RunningAvgSamplesPerSec=5.8670042558495545, CurrSamplesPerSec=5.322593108786849, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1620/5000 [5:25:18<10:20:45, 11.02s/it][2022-12-20 15:47:48,690] [INFO] [logging.py:68:log_dist] [Rank 0] step=1640, skipped=3, lr=[7.475555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:47:48,691] [INFO] [timer.py:197:stop] 0/3280, RunningAvgSamplesPerSec=5.867016854887844, CurrSamplesPerSec=5.3272296055775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1621/5000 [5:25:29<10:20:25, 11.02s/it][2022-12-20 15:47:59,677] [INFO] [timer.py:197:stop] 0/3282, RunningAvgSamplesPerSec=5.867031804107175, CurrSamplesPerSec=5.349545336749256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1622/5000 [5:25:40<10:19:27, 11.00s/it][2022-12-20 15:48:10,639] [INFO] [timer.py:197:stop] 0/3284, RunningAvgSamplesPerSec=5.867054659997944, CurrSamplesPerSec=5.357792938355677, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1623/5000 [5:25:51<10:19:35, 11.01s/it][2022-12-20 15:48:21,705] [INFO] [timer.py:197:stop] 0/3286, RunningAvgSamplesPerSec=5.867043729721724, CurrSamplesPerSec=5.318710686565158, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▏      | 1624/5000 [5:26:02<10:19:40, 11.01s/it][2022-12-20 15:48:32,687] [INFO] [timer.py:197:stop] 0/3288, RunningAvgSamplesPerSec=5.867060091493495, CurrSamplesPerSec=5.356515120458578, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 32%|███▎      | 1625/5000 [5:26:13<10:19:01, 11.00s/it]
                                                        
{'loss': 0.0003, 'learning_rate': 7.4666666666666675e-06, 'epoch': 39.63}

 32%|███▎      | 1625/5000 [5:26:13<10:19:01, 11.00s/it][2022-12-20 15:48:43,737] [INFO] [timer.py:197:stop] 0/3290, RunningAvgSamplesPerSec=5.867054133980714, CurrSamplesPerSec=5.298373006405052, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1626/5000 [5:26:24<10:19:46, 11.02s/it][2022-12-20 15:48:54,786] [INFO] [timer.py:197:stop] 0/3292, RunningAvgSamplesPerSec=5.867048637020643, CurrSamplesPerSec=5.30793937140706, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1627/5000 [5:26:35<10:20:09, 11.03s/it][2022-12-20 15:49:05,820] [INFO] [timer.py:197:stop] 0/3294, RunningAvgSamplesPerSec=5.867047904217109, CurrSamplesPerSec=5.326971657824347, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1628/5000 [5:26:46<10:21:55, 11.07s/it][2022-12-20 15:49:16,989] [INFO] [timer.py:197:stop] 0/3296, RunningAvgSamplesPerSec=5.867003141429918, CurrSamplesPerSec=5.307828958782243, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1629/5000 [5:26:57<10:21:15, 11.06s/it][2022-12-20 15:49:28,023] [INFO] [timer.py:197:stop] 0/3298, RunningAvgSamplesPerSec=5.867002203236517, CurrSamplesPerSec=5.310883372103201, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1630/5000 [5:27:08<10:21:13, 11.06s/it][2022-12-20 15:49:39,060] [INFO] [logging.py:68:log_dist] [Rank 0] step=1650, skipped=3, lr=[7.453333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:49:39,061] [INFO] [timer.py:197:stop] 0/3300, RunningAvgSamplesPerSec=5.866999972675358, CurrSamplesPerSec=5.335414850831249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1631/5000 [5:27:19<10:19:46, 11.04s/it][2022-12-20 15:49:50,063] [INFO] [timer.py:197:stop] 0/3302, RunningAvgSamplesPerSec=5.867009506783383, CurrSamplesPerSec=5.321571280188643, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1632/5000 [5:27:30<10:19:49, 11.04s/it][2022-12-20 15:50:01,131] [INFO] [timer.py:197:stop] 0/3304, RunningAvgSamplesPerSec=5.866998024627807, CurrSamplesPerSec=5.307377070572465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1633/5000 [5:27:41<10:20:17, 11.05s/it][2022-12-20 15:50:12,227] [INFO] [timer.py:197:stop] 0/3306, RunningAvgSamplesPerSec=5.8669770335560925, CurrSamplesPerSec=5.29417956597485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1634/5000 [5:27:52<10:23:34, 11.12s/it][2022-12-20 15:50:23,463] [INFO] [timer.py:197:stop] 0/3308, RunningAvgSamplesPerSec=5.86691056327345, CurrSamplesPerSec=5.31471859450834, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1635/5000 [5:28:03<10:21:14, 11.08s/it][2022-12-20 15:50:34,421] [INFO] [timer.py:197:stop] 0/3310, RunningAvgSamplesPerSec=5.866934121566561, CurrSamplesPerSec=5.339968554842867, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1636/5000 [5:28:14<10:19:31, 11.05s/it][2022-12-20 15:50:45,449] [INFO] [timer.py:197:stop] 0/3312, RunningAvgSamplesPerSec=5.86693569096524, CurrSamplesPerSec=5.304163182564428, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1637/5000 [5:28:26<10:24:54, 11.15s/it][2022-12-20 15:50:56,802] [INFO] [timer.py:197:stop] 0/3314, RunningAvgSamplesPerSec=5.866831364847706, CurrSamplesPerSec=5.328584241233403, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1638/5000 [5:28:37<10:22:25, 11.11s/it][2022-12-20 15:51:07,812] [INFO] [timer.py:197:stop] 0/3316, RunningAvgSamplesPerSec=5.866838339461951, CurrSamplesPerSec=5.330329256858428, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1639/5000 [5:28:48<10:19:52, 11.07s/it][2022-12-20 15:51:18,799] [INFO] [timer.py:197:stop] 0/3318, RunningAvgSamplesPerSec=5.866852634178701, CurrSamplesPerSec=5.312430506200632, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1640/5000 [5:28:59<10:24:12, 11.15s/it][2022-12-20 15:51:29,210] [INFO] [logging.py:68:log_dist] [Rank 0] step=1660, skipped=3, lr=[7.431111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:51:29,212] [INFO] [timer.py:197:stop] 0/3320, RunningAvgSamplesPerSec=5.867049929712862, CurrSamplesPerSec=6.260503581192741, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 15:51:40,187] [INFO] [timer.py:197:stop] 0/3322, RunningAvgSamplesPerSec=5.867067046209449, CurrSamplesPerSec=5.342708295807623, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1641/5000 [5:29:15<11:47:53, 12.64s/it][2022-12-20 15:51:51,207] [INFO] [timer.py:197:stop] 0/3324, RunningAvgSamplesPerSec=5.8670708194298005, CurrSamplesPerSec=5.32413714752481, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1642/5000 [5:29:26<11:20:23, 12.16s/it][2022-12-20 15:52:02,373] [INFO] [timer.py:197:stop] 0/3326, RunningAvgSamplesPerSec=5.867027155249485, CurrSamplesPerSec=5.305424950110097, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1643/5000 [5:29:37<11:03:33, 11.86s/it][2022-12-20 15:52:13,374] [INFO] [timer.py:197:stop] 0/3328, RunningAvgSamplesPerSec=5.86703695720371, CurrSamplesPerSec=5.3382299622077785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1644/5000 [5:29:48<10:48:57, 11.60s/it][2022-12-20 15:52:24,424] [INFO] [timer.py:197:stop] 0/3330, RunningAvgSamplesPerSec=5.867031115838578, CurrSamplesPerSec=5.296681664714217, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1645/5000 [5:30:00<10:39:29, 11.44s/it][2022-12-20 15:52:35,800] [INFO] [timer.py:197:stop] 0/3332, RunningAvgSamplesPerSec=5.866919742317112, CurrSamplesPerSec=5.30389048651986, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1646/5000 [5:30:11<10:38:16, 11.42s/it][2022-12-20 15:52:46,838] [INFO] [timer.py:197:stop] 0/3334, RunningAvgSamplesPerSec=5.8669172277168204, CurrSamplesPerSec=5.307308444026384, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1647/5000 [5:30:22<10:31:43, 11.30s/it][2022-12-20 15:52:57,861] [INFO] [timer.py:197:stop] 0/3336, RunningAvgSamplesPerSec=5.866919954219362, CurrSamplesPerSec=5.3232327426792905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1648/5000 [5:30:33<10:26:48, 11.22s/it][2022-12-20 15:53:08,974] [INFO] [timer.py:197:stop] 0/3338, RunningAvgSamplesPerSec=5.866893396781395, CurrSamplesPerSec=5.347613639809135, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1649/5000 [5:30:44<10:24:50, 11.19s/it][2022-12-20 15:53:20,009] [INFO] [logging.py:68:log_dist] [Rank 0] step=1670, skipped=3, lr=[7.40888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:53:20,010] [INFO] [timer.py:197:stop] 0/3340, RunningAvgSamplesPerSec=5.86689157467742, CurrSamplesPerSec=5.296723051952293, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1650/5000 [5:30:55<10:22:06, 11.14s/it]
                                                        
{'loss': 0.0003, 'learning_rate': 7.40888888888889e-06, 'epoch': 40.24}

 33%|███▎      | 1650/5000 [5:30:55<10:22:06, 11.14s/it][2022-12-20 15:53:30,986] [INFO] [timer.py:197:stop] 0/3342, RunningAvgSamplesPerSec=5.866910007618742, CurrSamplesPerSec=5.352008498600626, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1651/5000 [5:31:06<10:19:08, 11.09s/it][2022-12-20 15:53:42,174] [INFO] [timer.py:197:stop] 0/3344, RunningAvgSamplesPerSec=5.866859783396251, CurrSamplesPerSec=5.302444890400259, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1652/5000 [5:31:17<10:20:33, 11.12s/it][2022-12-20 15:53:53,226] [INFO] [timer.py:197:stop] 0/3346, RunningAvgSamplesPerSec=5.866853200118688, CurrSamplesPerSec=5.309491511651904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1653/5000 [5:31:28<10:19:12, 11.10s/it][2022-12-20 15:54:04,203] [INFO] [timer.py:197:stop] 0/3348, RunningAvgSamplesPerSec=5.866870879257763, CurrSamplesPerSec=5.344957837755523, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1654/5000 [5:31:39<10:16:57, 11.06s/it][2022-12-20 15:54:15,227] [INFO] [timer.py:197:stop] 0/3350, RunningAvgSamplesPerSec=5.866873430967599, CurrSamplesPerSec=5.364052050010181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1655/5000 [5:31:50<10:16:07, 11.05s/it][2022-12-20 15:54:26,293] [INFO] [timer.py:197:stop] 0/3352, RunningAvgSamplesPerSec=5.866862451849964, CurrSamplesPerSec=5.289379013554706, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1656/5000 [5:32:01<10:16:11, 11.06s/it][2022-12-20 15:54:37,252] [INFO] [timer.py:197:stop] 0/3354, RunningAvgSamplesPerSec=5.86688604176015, CurrSamplesPerSec=5.345163886884828, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1657/5000 [5:32:12<10:14:23, 11.03s/it][2022-12-20 15:54:48,526] [INFO] [timer.py:197:stop] 0/3356, RunningAvgSamplesPerSec=5.866808674888217, CurrSamplesPerSec=5.318992707918792, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1658/5000 [5:32:24<10:18:19, 11.10s/it][2022-12-20 15:54:59,570] [INFO] [timer.py:197:stop] 0/3358, RunningAvgSamplesPerSec=5.866805271501486, CurrSamplesPerSec=5.3131995737175775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1659/5000 [5:32:35<10:17:10, 11.08s/it][2022-12-20 15:55:10,556] [INFO] [logging.py:68:log_dist] [Rank 0] step=1680, skipped=3, lr=[7.386666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:55:10,558] [INFO] [timer.py:197:stop] 0/3360, RunningAvgSamplesPerSec=5.866819183279674, CurrSamplesPerSec=5.3356748896233634, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1660/5000 [5:32:46<10:15:24, 11.06s/it][2022-12-20 15:55:21,791] [INFO] [timer.py:197:stop] 0/3362, RunningAvgSamplesPerSec=5.866754658964823, CurrSamplesPerSec=5.2952943124057175, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1661/5000 [5:32:57<10:18:11, 11.11s/it][2022-12-20 15:55:32,774] [INFO] [timer.py:197:stop] 0/3364, RunningAvgSamplesPerSec=5.866770100969544, CurrSamplesPerSec=5.332643809995742, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1662/5000 [5:33:08<10:15:54, 11.07s/it][2022-12-20 15:55:43,828] [INFO] [timer.py:197:stop] 0/3366, RunningAvgSamplesPerSec=5.866763078028928, CurrSamplesPerSec=5.297346237995616, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1663/5000 [5:33:19<10:15:26, 11.07s/it][2022-12-20 15:55:54,886] [INFO] [timer.py:197:stop] 0/3368, RunningAvgSamplesPerSec=5.866759937331425, CurrSamplesPerSec=5.323342108019666, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1664/5000 [5:33:30<10:15:07, 11.06s/it][2022-12-20 15:56:05,862] [INFO] [timer.py:197:stop] 0/3370, RunningAvgSamplesPerSec=5.866777545837157, CurrSamplesPerSec=5.327645969034905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1665/5000 [5:33:41<10:13:29, 11.04s/it][2022-12-20 15:56:16,908] [INFO] [timer.py:197:stop] 0/3372, RunningAvgSamplesPerSec=5.866772870137784, CurrSamplesPerSec=5.3051374457817, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1666/5000 [5:33:52<10:13:27, 11.04s/it][2022-12-20 15:56:27,989] [INFO] [timer.py:197:stop] 0/3374, RunningAvgSamplesPerSec=5.866762662477689, CurrSamplesPerSec=5.304680145335362, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1667/5000 [5:34:03<10:13:56, 11.05s/it][2022-12-20 15:56:38,977] [INFO] [timer.py:197:stop] 0/3376, RunningAvgSamplesPerSec=5.8667765699747125, CurrSamplesPerSec=5.360375879657484, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1668/5000 [5:34:14<10:12:41, 11.03s/it][2022-12-20 15:56:49,944] [INFO] [timer.py:197:stop] 0/3378, RunningAvgSamplesPerSec=5.866797242380901, CurrSamplesPerSec=5.3272900789112745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1669/5000 [5:34:25<10:11:24, 11.01s/it][2022-12-20 15:57:01,048] [INFO] [logging.py:68:log_dist] [Rank 0] step=1690, skipped=3, lr=[7.364444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:57:01,049] [INFO] [timer.py:197:stop] 0/3380, RunningAvgSamplesPerSec=5.866773808021009, CurrSamplesPerSec=5.299749835095702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1670/5000 [5:34:36<10:12:45, 11.04s/it][2022-12-20 15:57:12,043] [INFO] [timer.py:197:stop] 0/3382, RunningAvgSamplesPerSec=5.86678574973704, CurrSamplesPerSec=5.361083514808536, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1671/5000 [5:34:47<10:11:48, 11.03s/it][2022-12-20 15:57:23,094] [INFO] [timer.py:197:stop] 0/3384, RunningAvgSamplesPerSec=5.8667798347917115, CurrSamplesPerSec=5.300819196193898, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1672/5000 [5:34:58<10:12:00, 11.03s/it][2022-12-20 15:57:34,169] [INFO] [timer.py:197:stop] 0/3386, RunningAvgSamplesPerSec=5.86676600368796, CurrSamplesPerSec=5.335087187143136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1673/5000 [5:35:09<10:12:31, 11.05s/it][2022-12-20 15:57:45,186] [INFO] [timer.py:197:stop] 0/3388, RunningAvgSamplesPerSec=5.866770974975621, CurrSamplesPerSec=5.344950387929622, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 33%|███▎      | 1674/5000 [5:35:20<10:11:50, 11.04s/it][2022-12-20 15:57:56,229] [INFO] [timer.py:197:stop] 0/3390, RunningAvgSamplesPerSec=5.866767455651793, CurrSamplesPerSec=5.303629345032969, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1675/5000 [5:35:31<10:11:45, 11.04s/it]
                                                        
{'loss': 0.0003, 'learning_rate': 7.353333333333334e-06, 'epoch': 40.84}

 34%|███▎      | 1675/5000 [5:35:31<10:11:45, 11.04s/it][2022-12-20 15:58:07,248] [INFO] [timer.py:197:stop] 0/3392, RunningAvgSamplesPerSec=5.866771905898879, CurrSamplesPerSec=5.333319134220203, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1676/5000 [5:35:42<10:11:13, 11.03s/it][2022-12-20 15:58:18,253] [INFO] [timer.py:197:stop] 0/3394, RunningAvgSamplesPerSec=5.866780427745965, CurrSamplesPerSec=5.350512881771158, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1677/5000 [5:35:53<10:10:34, 11.02s/it][2022-12-20 15:58:29,258] [INFO] [timer.py:197:stop] 0/3396, RunningAvgSamplesPerSec=5.866788779014053, CurrSamplesPerSec=5.3393452838247, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1678/5000 [5:36:04<10:10:04, 11.02s/it][2022-12-20 15:58:40,297] [INFO] [timer.py:197:stop] 0/3398, RunningAvgSamplesPerSec=5.866786278412618, CurrSamplesPerSec=5.344037622152068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1679/5000 [5:36:15<10:10:13, 11.02s/it][2022-12-20 15:58:51,283] [INFO] [logging.py:68:log_dist] [Rank 0] step=1700, skipped=3, lr=[7.342222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 15:58:51,284] [INFO] [timer.py:197:stop] 0/3400, RunningAvgSamplesPerSec=5.866800303620241, CurrSamplesPerSec=5.35636655168229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1680/5000 [5:36:26<10:09:24, 11.01s/it][2022-12-20 15:59:02,316] [INFO] [timer.py:197:stop] 0/3402, RunningAvgSamplesPerSec=5.866799861282996, CurrSamplesPerSec=5.322659387139845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1681/5000 [5:36:37<10:09:32, 11.02s/it][2022-12-20 15:59:12,454] [INFO] [timer.py:197:stop] 0/3404, RunningAvgSamplesPerSec=5.867079523543837, CurrSamplesPerSec=5.329056251916241, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1682/5000 [5:36:53<11:16:56, 12.24s/it][2022-12-20 15:59:23,497] [INFO] [timer.py:197:stop] 0/3406, RunningAvgSamplesPerSec=5.867075656315859, CurrSamplesPerSec=5.311733974108202, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1683/5000 [5:37:04<10:56:35, 11.88s/it][2022-12-20 15:59:34,504] [INFO] [timer.py:197:stop] 0/3408, RunningAvgSamplesPerSec=5.867082976361793, CurrSamplesPerSec=5.328866464373828, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1684/5000 [5:37:15<10:42:04, 11.62s/it][2022-12-20 15:59:45,475] [INFO] [timer.py:197:stop] 0/3410, RunningAvgSamplesPerSec=5.86710159900228, CurrSamplesPerSec=5.36688008726626, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1685/5000 [5:37:26<10:31:12, 11.42s/it][2022-12-20 15:59:56,476] [INFO] [timer.py:197:stop] 0/3412, RunningAvgSamplesPerSec=5.8671108087659904, CurrSamplesPerSec=5.342346988245683, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1686/5000 [5:37:36<10:23:19, 11.29s/it][2022-12-20 16:00:07,467] [INFO] [timer.py:197:stop] 0/3414, RunningAvgSamplesPerSec=5.867123191414505, CurrSamplesPerSec=5.314206827347002, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▎      | 1687/5000 [5:37:48<10:19:03, 11.21s/it][2022-12-20 16:00:18,514] [INFO] [timer.py:197:stop] 0/3416, RunningAvgSamplesPerSec=5.867117852702607, CurrSamplesPerSec=5.307745417681593, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1688/5000 [5:37:59<10:16:06, 11.16s/it][2022-12-20 16:00:29,511] [INFO] [timer.py:197:stop] 0/3418, RunningAvgSamplesPerSec=5.867128167001346, CurrSamplesPerSec=5.34911914792693, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1689/5000 [5:38:09<10:11:59, 11.09s/it][2022-12-20 16:00:40,519] [INFO] [logging.py:68:log_dist] [Rank 0] step=1710, skipped=3, lr=[7.32e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:00:40,520] [INFO] [timer.py:197:stop] 0/3420, RunningAvgSamplesPerSec=5.867135298763588, CurrSamplesPerSec=5.275708508357985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1690/5000 [5:38:21<10:12:10, 11.10s/it][2022-12-20 16:00:51,574] [INFO] [timer.py:197:stop] 0/3422, RunningAvgSamplesPerSec=5.867132899307006, CurrSamplesPerSec=5.32589425607541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1691/5000 [5:38:32<10:10:25, 11.07s/it][2022-12-20 16:01:02,597] [INFO] [timer.py:197:stop] 0/3424, RunningAvgSamplesPerSec=5.867136115388486, CurrSamplesPerSec=5.309805955786789, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1692/5000 [5:38:43<10:10:03, 11.07s/it][2022-12-20 16:01:13,684] [INFO] [timer.py:197:stop] 0/3426, RunningAvgSamplesPerSec=5.867117980474201, CurrSamplesPerSec=5.280994546696535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1693/5000 [5:38:54<10:09:47, 11.06s/it][2022-12-20 16:01:24,696] [INFO] [timer.py:197:stop] 0/3428, RunningAvgSamplesPerSec=5.867123741437511, CurrSamplesPerSec=5.323413050102763, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1694/5000 [5:39:05<10:09:17, 11.06s/it][2022-12-20 16:01:35,741] [INFO] [timer.py:197:stop] 0/3430, RunningAvgSamplesPerSec=5.867119110127389, CurrSamplesPerSec=5.3240250040629, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1695/5000 [5:39:16<10:07:56, 11.04s/it][2022-12-20 16:01:46,836] [INFO] [timer.py:197:stop] 0/3432, RunningAvgSamplesPerSec=5.867099109953734, CurrSamplesPerSec=5.230543069489109, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1696/5000 [5:39:27<10:08:59, 11.06s/it][2022-12-20 16:01:57,811] [INFO] [timer.py:197:stop] 0/3434, RunningAvgSamplesPerSec=5.867116230895215, CurrSamplesPerSec=5.3497376662982665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1697/5000 [5:39:38<10:07:22, 11.03s/it][2022-12-20 16:02:08,843] [INFO] [timer.py:197:stop] 0/3436, RunningAvgSamplesPerSec=5.867115586251071, CurrSamplesPerSec=5.2971902709387795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1698/5000 [5:39:49<10:07:35, 11.04s/it][2022-12-20 16:02:19,912] [INFO] [timer.py:197:stop] 0/3438, RunningAvgSamplesPerSec=5.867103056385145, CurrSamplesPerSec=5.286448180075338, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1699/5000 [5:40:00<10:07:48, 11.05s/it][2022-12-20 16:02:30,928] [INFO] [logging.py:68:log_dist] [Rank 0] step=1720, skipped=3, lr=[7.297777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:02:30,930] [INFO] [timer.py:197:stop] 0/3440, RunningAvgSamplesPerSec=5.867106655356168, CurrSamplesPerSec=5.327902713120213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1700/5000 [5:40:11<10:07:06, 11.04s/it]
                                                        
{'loss': 0.0003, 'learning_rate': 7.297777777777778e-06, 'epoch': 41.46}

 34%|███▍      | 1700/5000 [5:40:11<10:07:06, 11.04s/it][2022-12-20 16:02:41,971] [INFO] [timer.py:197:stop] 0/3442, RunningAvgSamplesPerSec=5.867103893097264, CurrSamplesPerSec=5.3084610597104795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1701/5000 [5:40:22<10:06:56, 11.04s/it][2022-12-20 16:02:53,070] [INFO] [timer.py:197:stop] 0/3444, RunningAvgSamplesPerSec=5.8670821997854725, CurrSamplesPerSec=5.2537850929094505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1702/5000 [5:40:33<10:08:01, 11.06s/it][2022-12-20 16:03:04,157] [INFO] [timer.py:197:stop] 0/3446, RunningAvgSamplesPerSec=5.867065006970918, CurrSamplesPerSec=5.280127172099811, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1703/5000 [5:40:44<10:07:48, 11.06s/it][2022-12-20 16:03:15,219] [INFO] [timer.py:197:stop] 0/3448, RunningAvgSamplesPerSec=5.867056278394256, CurrSamplesPerSec=5.279830979608147, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1704/5000 [5:40:55<10:07:48, 11.06s/it][2022-12-20 16:03:26,284] [INFO] [timer.py:197:stop] 0/3450, RunningAvgSamplesPerSec=5.867045188108756, CurrSamplesPerSec=5.282643239808394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1705/5000 [5:41:06<10:07:00, 11.05s/it][2022-12-20 16:03:37,235] [INFO] [timer.py:197:stop] 0/3452, RunningAvgSamplesPerSec=5.867069530550082, CurrSamplesPerSec=5.350043033856596, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1706/5000 [5:41:17<10:05:34, 11.03s/it][2022-12-20 16:03:48,230] [INFO] [timer.py:197:stop] 0/3454, RunningAvgSamplesPerSec=5.867080637491387, CurrSamplesPerSec=5.3358267673963145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1707/5000 [5:41:28<10:05:17, 11.03s/it][2022-12-20 16:03:59,540] [INFO] [timer.py:197:stop] 0/3456, RunningAvgSamplesPerSec=5.866993210407371, CurrSamplesPerSec=5.092445190596961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1708/5000 [5:41:40<10:09:26, 11.11s/it][2022-12-20 16:04:10,533] [INFO] [timer.py:197:stop] 0/3458, RunningAvgSamplesPerSec=5.867004910643209, CurrSamplesPerSec=5.346908064809192, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1709/5000 [5:41:51<10:06:35, 11.06s/it][2022-12-20 16:04:21,514] [INFO] [logging.py:68:log_dist] [Rank 0] step=1730, skipped=3, lr=[7.2755555555555554e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:04:21,516] [INFO] [timer.py:197:stop] 0/3460, RunningAvgSamplesPerSec=5.86702412783182, CurrSamplesPerSec=5.314215454183853, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1710/5000 [5:42:02<10:06:39, 11.06s/it][2022-12-20 16:04:32,577] [INFO] [timer.py:197:stop] 0/3462, RunningAvgSamplesPerSec=5.867015131148, CurrSamplesPerSec=5.328174922711971, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1711/5000 [5:42:13<10:05:08, 11.04s/it][2022-12-20 16:04:43,604] [INFO] [timer.py:197:stop] 0/3464, RunningAvgSamplesPerSec=5.867015909501113, CurrSamplesPerSec=5.286568740992702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1712/5000 [5:42:24<10:05:34, 11.05s/it][2022-12-20 16:04:54,579] [INFO] [timer.py:197:stop] 0/3466, RunningAvgSamplesPerSec=5.867036179172931, CurrSamplesPerSec=5.377537100004852, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1713/5000 [5:42:35<10:04:19, 11.03s/it][2022-12-20 16:05:05,663] [INFO] [timer.py:197:stop] 0/3468, RunningAvgSamplesPerSec=5.867019779969231, CurrSamplesPerSec=5.291153508336211, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1714/5000 [5:42:46<10:04:13, 11.03s/it][2022-12-20 16:05:16,616] [INFO] [timer.py:197:stop] 0/3470, RunningAvgSamplesPerSec=5.86704756131417, CurrSamplesPerSec=5.363740151123999, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1715/5000 [5:42:57<10:03:08, 11.02s/it][2022-12-20 16:05:27,602] [INFO] [timer.py:197:stop] 0/3472, RunningAvgSamplesPerSec=5.867061725803188, CurrSamplesPerSec=5.357493314895133, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1716/5000 [5:43:08<10:02:26, 11.01s/it][2022-12-20 16:05:38,639] [INFO] [timer.py:197:stop] 0/3474, RunningAvgSamplesPerSec=5.8670598794668445, CurrSamplesPerSec=5.310546527478033, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1717/5000 [5:43:19<10:03:06, 11.02s/it][2022-12-20 16:05:49,655] [INFO] [timer.py:197:stop] 0/3476, RunningAvgSamplesPerSec=5.867064609861847, CurrSamplesPerSec=5.348741838816351, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1718/5000 [5:43:30<10:02:26, 11.01s/it][2022-12-20 16:06:00,694] [INFO] [timer.py:197:stop] 0/3478, RunningAvgSamplesPerSec=5.8670666838942465, CurrSamplesPerSec=5.307431007666168, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1719/5000 [5:43:41<10:02:50, 11.02s/it][2022-12-20 16:06:11,788] [INFO] [logging.py:68:log_dist] [Rank 0] step=1740, skipped=3, lr=[7.253333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:06:11,790] [INFO] [timer.py:197:stop] 0/3480, RunningAvgSamplesPerSec=5.867046585928956, CurrSamplesPerSec=5.266333286144012, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1720/5000 [5:43:52<10:04:07, 11.05s/it][2022-12-20 16:06:22,793] [INFO] [timer.py:197:stop] 0/3482, RunningAvgSamplesPerSec=5.867055216136801, CurrSamplesPerSec=5.364685176942479, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1721/5000 [5:44:03<10:02:28, 11.02s/it][2022-12-20 16:06:33,820] [INFO] [timer.py:197:stop] 0/3484, RunningAvgSamplesPerSec=5.867056337121522, CurrSamplesPerSec=5.306165139256972, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1722/5000 [5:44:14<10:02:51, 11.03s/it][2022-12-20 16:06:44,027] [INFO] [timer.py:197:stop] 0/3486, RunningAvgSamplesPerSec=5.867307515851994, CurrSamplesPerSec=6.167097632570031, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 16:06:55,055] [INFO] [timer.py:197:stop] 0/3488, RunningAvgSamplesPerSec=5.867307366416999, CurrSamplesPerSec=5.319168090538513, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1723/5000 [5:44:30<11:28:27, 12.61s/it][2022-12-20 16:07:06,023] [INFO] [timer.py:197:stop] 0/3490, RunningAvgSamplesPerSec=5.867326994845612, CurrSamplesPerSec=5.376976974875648, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1724/5000 [5:44:41<11:01:25, 12.11s/it][2022-12-20 16:07:17,057] [INFO] [timer.py:197:stop] 0/3492, RunningAvgSamplesPerSec=5.867326367003451, CurrSamplesPerSec=5.335825070390194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 34%|███▍      | 1725/5000 [5:44:52<10:43:32, 11.79s/it]
                                                        
{'loss': 0.0003, 'learning_rate': 7.24e-06, 'epoch': 42.07}

 34%|███▍      | 1725/5000 [5:44:52<10:43:32, 11.79s/it][2022-12-20 16:07:28,070] [INFO] [timer.py:197:stop] 0/3494, RunningAvgSamplesPerSec=5.8673325273995465, CurrSamplesPerSec=5.316817618065792, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1726/5000 [5:45:03<10:30:38, 11.56s/it][2022-12-20 16:07:39,087] [INFO] [timer.py:197:stop] 0/3496, RunningAvgSamplesPerSec=5.867337238415593, CurrSamplesPerSec=5.331703048747376, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1727/5000 [5:45:14<10:21:35, 11.40s/it][2022-12-20 16:07:50,075] [INFO] [timer.py:197:stop] 0/3498, RunningAvgSamplesPerSec=5.86735077812428, CurrSamplesPerSec=5.357768342733404, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1728/5000 [5:45:25<10:14:44, 11.27s/it][2022-12-20 16:08:01,073] [INFO] [logging.py:68:log_dist] [Rank 0] step=1750, skipped=3, lr=[7.231111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:08:01,074] [INFO] [timer.py:197:stop] 0/3500, RunningAvgSamplesPerSec=5.867360417984349, CurrSamplesPerSec=5.352733565509722, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1729/5000 [5:45:36<10:10:05, 11.19s/it][2022-12-20 16:08:12,099] [INFO] [timer.py:197:stop] 0/3502, RunningAvgSamplesPerSec=5.867362219554452, CurrSamplesPerSec=5.323440920724167, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1730/5000 [5:45:47<10:07:11, 11.14s/it][2022-12-20 16:08:23,169] [INFO] [timer.py:197:stop] 0/3504, RunningAvgSamplesPerSec=5.867350506540717, CurrSamplesPerSec=5.301670974989135, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1731/5000 [5:45:58<10:05:50, 11.12s/it][2022-12-20 16:08:34,166] [INFO] [timer.py:197:stop] 0/3506, RunningAvgSamplesPerSec=5.867361741987509, CurrSamplesPerSec=5.337939103829883, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1732/5000 [5:46:09<10:03:38, 11.08s/it][2022-12-20 16:08:45,214] [INFO] [timer.py:197:stop] 0/3508, RunningAvgSamplesPerSec=5.867356261126641, CurrSamplesPerSec=5.307091243261932, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1733/5000 [5:46:20<10:02:53, 11.07s/it][2022-12-20 16:08:56,248] [INFO] [timer.py:197:stop] 0/3510, RunningAvgSamplesPerSec=5.86736042817816, CurrSamplesPerSec=5.325260109754112, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1734/5000 [5:46:31<10:02:04, 11.06s/it][2022-12-20 16:09:07,316] [INFO] [timer.py:197:stop] 0/3512, RunningAvgSamplesPerSec=5.8673541519529815, CurrSamplesPerSec=5.310776409323368, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1735/5000 [5:46:42<10:02:00, 11.06s/it][2022-12-20 16:09:18,319] [INFO] [timer.py:197:stop] 0/3514, RunningAvgSamplesPerSec=5.8673628485629905, CurrSamplesPerSec=5.320781858165661, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1736/5000 [5:46:53<10:00:51, 11.05s/it][2022-12-20 16:09:29,364] [INFO] [timer.py:197:stop] 0/3516, RunningAvgSamplesPerSec=5.8673638928230805, CurrSamplesPerSec=5.335972926098386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1737/5000 [5:47:04<10:00:39, 11.04s/it][2022-12-20 16:09:40,436] [INFO] [timer.py:197:stop] 0/3518, RunningAvgSamplesPerSec=5.867351391670739, CurrSamplesPerSec=5.257804771763849, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1738/5000 [5:47:16<10:00:55, 11.05s/it][2022-12-20 16:09:51,488] [INFO] [logging.py:68:log_dist] [Rank 0] step=1760, skipped=3, lr=[7.20888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:09:51,489] [INFO] [timer.py:197:stop] 0/3520, RunningAvgSamplesPerSec=5.867344446400452, CurrSamplesPerSec=5.302402785266356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1739/5000 [5:47:27<10:00:44, 11.05s/it][2022-12-20 16:10:02,530] [INFO] [timer.py:197:stop] 0/3522, RunningAvgSamplesPerSec=5.8673469009264805, CurrSamplesPerSec=5.28829905218775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1740/5000 [5:47:38<10:00:21, 11.05s/it][2022-12-20 16:10:13,599] [INFO] [timer.py:197:stop] 0/3524, RunningAvgSamplesPerSec=5.867340676590109, CurrSamplesPerSec=5.315060388653837, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1741/5000 [5:47:49<10:00:29, 11.06s/it][2022-12-20 16:10:24,616] [INFO] [timer.py:197:stop] 0/3526, RunningAvgSamplesPerSec=5.867344985008762, CurrSamplesPerSec=5.307992690174752, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1742/5000 [5:48:00<9:59:40, 11.04s/it] [2022-12-20 16:10:35,625] [INFO] [timer.py:197:stop] 0/3528, RunningAvgSamplesPerSec=5.867351714819881, CurrSamplesPerSec=5.316517926816032, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1743/5000 [5:48:11<9:58:55, 11.03s/it][2022-12-20 16:10:46,671] [INFO] [timer.py:197:stop] 0/3530, RunningAvgSamplesPerSec=5.867352419472016, CurrSamplesPerSec=5.300346313320061, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1744/5000 [5:48:22<9:58:57, 11.04s/it][2022-12-20 16:10:57,726] [INFO] [timer.py:197:stop] 0/3532, RunningAvgSamplesPerSec=5.867345420424728, CurrSamplesPerSec=5.326640591005977, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1745/5000 [5:48:33<9:59:03, 11.04s/it][2022-12-20 16:11:08,736] [INFO] [timer.py:197:stop] 0/3534, RunningAvgSamplesPerSec=5.867351700807187, CurrSamplesPerSec=5.343238330338344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1746/5000 [5:48:44<9:58:20, 11.03s/it][2022-12-20 16:11:19,756] [INFO] [timer.py:197:stop] 0/3536, RunningAvgSamplesPerSec=5.867360731575583, CurrSamplesPerSec=5.356464242741094, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1747/5000 [5:48:55<9:57:56, 11.03s/it][2022-12-20 16:11:30,781] [INFO] [timer.py:197:stop] 0/3538, RunningAvgSamplesPerSec=5.867362170290342, CurrSamplesPerSec=5.313764160924343, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1748/5000 [5:49:06<9:57:43, 11.03s/it][2022-12-20 16:11:41,737] [INFO] [logging.py:68:log_dist] [Rank 0] step=1770, skipped=3, lr=[7.186666666666668e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:11:41,738] [INFO] [timer.py:197:stop] 0/3540, RunningAvgSamplesPerSec=5.867386589578564, CurrSamplesPerSec=5.3594370794750255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▍      | 1749/5000 [5:49:17<9:56:22, 11.01s/it][2022-12-20 16:11:52,799] [INFO] [timer.py:197:stop] 0/3542, RunningAvgSamplesPerSec=5.86737765083096, CurrSamplesPerSec=5.320919810811181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1750/5000 [5:49:28<9:57:04, 11.02s/it]
                                                       
{'loss': 0.0003, 'learning_rate': 7.184444444444445e-06, 'epoch': 42.67}

 35%|███▌      | 1750/5000 [5:49:28<9:57:04, 11.02s/it][2022-12-20 16:12:03,837] [INFO] [timer.py:197:stop] 0/3544, RunningAvgSamplesPerSec=5.86737590577841, CurrSamplesPerSec=5.314646831740549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1751/5000 [5:49:39<9:57:08, 11.03s/it][2022-12-20 16:12:14,821] [INFO] [timer.py:197:stop] 0/3546, RunningAvgSamplesPerSec=5.867392357996859, CurrSamplesPerSec=5.3599719369450245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1752/5000 [5:49:50<9:56:13, 11.01s/it][2022-12-20 16:12:25,971] [INFO] [timer.py:197:stop] 0/3548, RunningAvgSamplesPerSec=5.867355870596253, CurrSamplesPerSec=5.220298132220232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1753/5000 [5:50:01<9:58:15, 11.06s/it][2022-12-20 16:12:37,030] [INFO] [timer.py:197:stop] 0/3550, RunningAvgSamplesPerSec=5.867347242026122, CurrSamplesPerSec=5.316565942536702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1754/5000 [5:50:12<9:58:09, 11.06s/it][2022-12-20 16:12:48,091] [INFO] [timer.py:197:stop] 0/3552, RunningAvgSamplesPerSec=5.867343450082506, CurrSamplesPerSec=5.310175690583778, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1755/5000 [5:50:23<9:58:01, 11.06s/it][2022-12-20 16:12:59,086] [INFO] [timer.py:197:stop] 0/3554, RunningAvgSamplesPerSec=5.867354390696369, CurrSamplesPerSec=5.327222627973488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1756/5000 [5:50:34<9:56:49, 11.04s/it][2022-12-20 16:13:10,110] [INFO] [timer.py:197:stop] 0/3556, RunningAvgSamplesPerSec=5.867356406948019, CurrSamplesPerSec=5.305559800584736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1757/5000 [5:50:45<9:56:24, 11.03s/it][2022-12-20 16:13:21,154] [INFO] [timer.py:197:stop] 0/3558, RunningAvgSamplesPerSec=5.8673526526614355, CurrSamplesPerSec=5.322515223277781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1758/5000 [5:50:56<9:56:22, 11.04s/it][2022-12-20 16:13:32,165] [INFO] [logging.py:68:log_dist] [Rank 0] step=1780, skipped=3, lr=[7.164444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:13:32,167] [INFO] [timer.py:197:stop] 0/3560, RunningAvgSamplesPerSec=5.867358384820443, CurrSamplesPerSec=5.319745965345125, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1759/5000 [5:51:07<9:55:47, 11.03s/it][2022-12-20 16:13:43,236] [INFO] [timer.py:197:stop] 0/3562, RunningAvgSamplesPerSec=5.867346569984162, CurrSamplesPerSec=5.313920895116661, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1760/5000 [5:51:18<9:56:15, 11.04s/it][2022-12-20 16:13:54,264] [INFO] [timer.py:197:stop] 0/3564, RunningAvgSamplesPerSec=5.867351644557277, CurrSamplesPerSec=5.337596695867524, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1761/5000 [5:51:29<9:55:50, 11.04s/it][2022-12-20 16:14:05,357] [INFO] [timer.py:197:stop] 0/3566, RunningAvgSamplesPerSec=5.867333113542462, CurrSamplesPerSec=5.298407099433031, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1762/5000 [5:51:40<9:56:33, 11.05s/it][2022-12-20 16:14:16,411] [INFO] [timer.py:197:stop] 0/3568, RunningAvgSamplesPerSec=5.8673263654851615, CurrSamplesPerSec=5.291144539036019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1763/5000 [5:51:52<9:56:23, 11.05s/it][2022-12-20 16:14:26,551] [INFO] [timer.py:197:stop] 0/3570, RunningAvgSamplesPerSec=5.867596458832462, CurrSamplesPerSec=5.317581425407733, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1764/5000 [5:52:07<11:00:42, 12.25s/it][2022-12-20 16:14:37,623] [INFO] [timer.py:197:stop] 0/3572, RunningAvgSamplesPerSec=5.867583405405419, CurrSamplesPerSec=5.240314497593815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1765/5000 [5:52:18<10:42:04, 11.91s/it][2022-12-20 16:14:48,598] [INFO] [timer.py:197:stop] 0/3574, RunningAvgSamplesPerSec=5.8676001486299985, CurrSamplesPerSec=5.359767741063648, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1766/5000 [5:52:29<10:27:05, 11.63s/it][2022-12-20 16:14:59,602] [INFO] [timer.py:197:stop] 0/3576, RunningAvgSamplesPerSec=5.867607874296465, CurrSamplesPerSec=5.351342088439861, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1767/5000 [5:52:40<10:16:05, 11.43s/it][2022-12-20 16:15:10,639] [INFO] [timer.py:197:stop] 0/3578, RunningAvgSamplesPerSec=5.867605761137408, CurrSamplesPerSec=5.288008817437093, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1768/5000 [5:52:51<10:09:29, 11.31s/it][2022-12-20 16:15:21,644] [INFO] [logging.py:68:log_dist] [Rank 0] step=1790, skipped=3, lr=[7.142222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:15:21,645] [INFO] [timer.py:197:stop] 0/3580, RunningAvgSamplesPerSec=5.8676176118390355, CurrSamplesPerSec=5.3152957133762815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1769/5000 [5:53:02<10:04:38, 11.23s/it][2022-12-20 16:15:32,639] [INFO] [timer.py:197:stop] 0/3582, RunningAvgSamplesPerSec=5.867628478955337, CurrSamplesPerSec=5.3450721419737635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1770/5000 [5:53:13<10:00:41, 11.16s/it][2022-12-20 16:15:43,718] [INFO] [timer.py:197:stop] 0/3584, RunningAvgSamplesPerSec=5.867613445663599, CurrSamplesPerSec=5.270697754492382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1771/5000 [5:53:24<9:59:25, 11.14s/it] [2022-12-20 16:15:54,720] [INFO] [timer.py:197:stop] 0/3586, RunningAvgSamplesPerSec=5.86762205608351, CurrSamplesPerSec=5.3497896957368205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1772/5000 [5:53:35<9:57:02, 11.10s/it][2022-12-20 16:16:05,717] [INFO] [timer.py:197:stop] 0/3588, RunningAvgSamplesPerSec=5.867635507783562, CurrSamplesPerSec=5.354506398867133, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1773/5000 [5:53:46<9:55:00, 11.06s/it][2022-12-20 16:16:16,702] [INFO] [timer.py:197:stop] 0/3590, RunningAvgSamplesPerSec=5.867649107553814, CurrSamplesPerSec=5.352905843137422, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 35%|███▌      | 1774/5000 [5:53:57<9:54:00, 11.05s/it][2022-12-20 16:16:27,718] [INFO] [timer.py:197:stop] 0/3592, RunningAvgSamplesPerSec=5.867653749220247, CurrSamplesPerSec=5.3501549964784125, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1775/5000 [5:54:08<9:52:18, 11.02s/it]
                                                       
{'loss': 0.0003, 'learning_rate': 7.12888888888889e-06, 'epoch': 43.29}

 36%|███▌      | 1775/5000 [5:54:08<9:52:18, 11.02s/it][2022-12-20 16:16:38,700] [INFO] [timer.py:197:stop] 0/3594, RunningAvgSamplesPerSec=5.867668523604353, CurrSamplesPerSec=5.325266659643956, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1776/5000 [5:54:19<9:52:15, 11.02s/it][2022-12-20 16:16:49,687] [INFO] [timer.py:197:stop] 0/3596, RunningAvgSamplesPerSec=5.867681340865713, CurrSamplesPerSec=5.360628294483764, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1777/5000 [5:54:30<9:50:59, 11.00s/it][2022-12-20 16:17:00,651] [INFO] [timer.py:197:stop] 0/3598, RunningAvgSamplesPerSec=5.867701318782148, CurrSamplesPerSec=5.352783304999633, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1778/5000 [5:54:41<9:50:16, 10.99s/it][2022-12-20 16:17:11,574] [INFO] [logging.py:68:log_dist] [Rank 0] step=1800, skipped=3, lr=[7.1200000000000004e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:17:11,576] [INFO] [timer.py:197:stop] 0/3600, RunningAvgSamplesPerSec=5.867733005480635, CurrSamplesPerSec=5.393124441521702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1779/5000 [5:54:52<9:48:56, 10.97s/it][2022-12-20 16:17:22,589] [INFO] [timer.py:197:stop] 0/3602, RunningAvgSamplesPerSec=5.867737649483979, CurrSamplesPerSec=5.309725923385553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1780/5000 [5:55:03<9:48:51, 10.97s/it][2022-12-20 16:17:33,525] [INFO] [timer.py:197:stop] 0/3604, RunningAvgSamplesPerSec=5.867765906315899, CurrSamplesPerSec=5.347261894213736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1781/5000 [5:55:14<9:49:00, 10.98s/it][2022-12-20 16:17:44,546] [INFO] [timer.py:197:stop] 0/3606, RunningAvgSamplesPerSec=5.867768586479001, CurrSamplesPerSec=5.321789667829376, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1782/5000 [5:55:25<9:48:56, 10.98s/it][2022-12-20 16:17:55,491] [INFO] [timer.py:197:stop] 0/3608, RunningAvgSamplesPerSec=5.867793552327495, CurrSamplesPerSec=5.357194152600634, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1783/5000 [5:55:36<9:48:50, 10.98s/it][2022-12-20 16:18:06,530] [INFO] [timer.py:197:stop] 0/3610, RunningAvgSamplesPerSec=5.8677905521950295, CurrSamplesPerSec=5.31032107768662, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1784/5000 [5:55:47<9:49:33, 11.00s/it][2022-12-20 16:18:17,517] [INFO] [timer.py:197:stop] 0/3612, RunningAvgSamplesPerSec=5.867802799163721, CurrSamplesPerSec=5.355282997103622, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1785/5000 [5:55:58<9:49:15, 11.00s/it][2022-12-20 16:18:28,566] [INFO] [timer.py:197:stop] 0/3614, RunningAvgSamplesPerSec=5.867796786673767, CurrSamplesPerSec=5.305290316195042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1786/5000 [5:56:09<9:49:48, 11.01s/it][2022-12-20 16:18:39,551] [INFO] [timer.py:197:stop] 0/3616, RunningAvgSamplesPerSec=5.8678097220916206, CurrSamplesPerSec=5.355854640394681, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1787/5000 [5:56:20<9:49:04, 11.00s/it][2022-12-20 16:18:50,588] [INFO] [timer.py:197:stop] 0/3618, RunningAvgSamplesPerSec=5.867807372672665, CurrSamplesPerSec=5.30306858553917, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1788/5000 [5:56:31<9:49:12, 11.01s/it][2022-12-20 16:19:01,587] [INFO] [logging.py:68:log_dist] [Rank 0] step=1810, skipped=3, lr=[7.097777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:19:01,589] [INFO] [timer.py:197:stop] 0/3620, RunningAvgSamplesPerSec=5.8678156441538025, CurrSamplesPerSec=5.320339572001522, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1789/5000 [5:56:42<9:49:17, 11.01s/it][2022-12-20 16:19:12,587] [INFO] [timer.py:197:stop] 0/3622, RunningAvgSamplesPerSec=5.867824806299585, CurrSamplesPerSec=5.341856037050231, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1790/5000 [5:56:53<9:48:20, 11.00s/it][2022-12-20 16:19:23,570] [INFO] [timer.py:197:stop] 0/3624, RunningAvgSamplesPerSec=5.867838705663901, CurrSamplesPerSec=5.325347372512371, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1791/5000 [5:57:04<9:48:47, 11.01s/it][2022-12-20 16:19:34,614] [INFO] [timer.py:197:stop] 0/3626, RunningAvgSamplesPerSec=5.867834194811847, CurrSamplesPerSec=5.31771499837736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1792/5000 [5:57:15<9:48:45, 11.01s/it][2022-12-20 16:19:45,633] [INFO] [timer.py:197:stop] 0/3628, RunningAvgSamplesPerSec=5.867837066394823, CurrSamplesPerSec=5.316711047945764, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1793/5000 [5:57:26<9:48:51, 11.02s/it][2022-12-20 16:19:56,676] [INFO] [timer.py:197:stop] 0/3630, RunningAvgSamplesPerSec=5.86783311217931, CurrSamplesPerSec=5.305689414500028, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1794/5000 [5:57:37<9:49:32, 11.03s/it][2022-12-20 16:20:07,752] [INFO] [timer.py:197:stop] 0/3632, RunningAvgSamplesPerSec=5.8678192171607195, CurrSamplesPerSec=5.30120422158682, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1795/5000 [5:57:48<9:49:13, 11.03s/it][2022-12-20 16:20:18,776] [INFO] [timer.py:197:stop] 0/3634, RunningAvgSamplesPerSec=5.867826088633927, CurrSamplesPerSec=5.317065525959461, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1796/5000 [5:57:59<9:49:56, 11.05s/it][2022-12-20 16:20:29,866] [INFO] [timer.py:197:stop] 0/3636, RunningAvgSamplesPerSec=5.8678083359500075, CurrSamplesPerSec=5.29959895784635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1797/5000 [5:58:10<9:49:12, 11.04s/it][2022-12-20 16:20:40,868] [INFO] [timer.py:197:stop] 0/3638, RunningAvgSamplesPerSec=5.867816582888875, CurrSamplesPerSec=5.309409388422586, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1798/5000 [5:58:21<9:47:55, 11.02s/it][2022-12-20 16:20:51,786] [INFO] [logging.py:68:log_dist] [Rank 0] step=1820, skipped=3, lr=[7.0755555555555565e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:20:51,787] [INFO] [timer.py:197:stop] 0/3640, RunningAvgSamplesPerSec=5.867848963113875, CurrSamplesPerSec=5.352412949437624, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1799/5000 [5:58:32<9:46:52, 11.00s/it][2022-12-20 16:21:02,768] [INFO] [timer.py:197:stop] 0/3642, RunningAvgSamplesPerSec=5.867863518411578, CurrSamplesPerSec=5.337413516093198, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1800/5000 [5:58:43<9:46:27, 11.00s/it]
                                                       
{'loss': 0.0003, 'learning_rate': 7.073333333333334e-06, 'epoch': 43.89}

 36%|███▌      | 1800/5000 [5:58:43<9:46:27, 11.00s/it][2022-12-20 16:21:13,797] [INFO] [timer.py:197:stop] 0/3644, RunningAvgSamplesPerSec=5.867863865032752, CurrSamplesPerSec=5.3000020138927, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1801/5000 [5:58:54<9:47:29, 11.02s/it][2022-12-20 16:21:24,902] [INFO] [timer.py:197:stop] 0/3646, RunningAvgSamplesPerSec=5.867855906001333, CurrSamplesPerSec=5.299130267792619, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1802/5000 [5:59:05<9:48:35, 11.04s/it][2022-12-20 16:21:35,952] [INFO] [timer.py:197:stop] 0/3648, RunningAvgSamplesPerSec=5.867850075133944, CurrSamplesPerSec=5.313642987537273, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1803/5000 [5:59:16<9:48:26, 11.04s/it][2022-12-20 16:21:46,988] [INFO] [timer.py:197:stop] 0/3650, RunningAvgSamplesPerSec=5.867848148493058, CurrSamplesPerSec=5.321127175109992, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1804/5000 [5:59:27<9:47:14, 11.02s/it][2022-12-20 16:21:57,042] [INFO] [timer.py:197:stop] 0/3652, RunningAvgSamplesPerSec=5.868132621584865, CurrSamplesPerSec=6.277680904409208, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 16:22:08,095] [INFO] [timer.py:197:stop] 0/3654, RunningAvgSamplesPerSec=5.868125109190584, CurrSamplesPerSec=5.301034208916655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1805/5000 [5:59:43<11:09:47, 12.58s/it][2022-12-20 16:22:19,093] [INFO] [timer.py:197:stop] 0/3656, RunningAvgSamplesPerSec=5.868134462380748, CurrSamplesPerSec=5.355363340341336, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1806/5000 [5:59:54<10:44:21, 12.10s/it][2022-12-20 16:22:30,155] [INFO] [timer.py:197:stop] 0/3658, RunningAvgSamplesPerSec=5.868125109133783, CurrSamplesPerSec=5.30359707090439, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1807/5000 [6:00:05<10:27:30, 11.79s/it][2022-12-20 16:22:41,190] [INFO] [logging.py:68:log_dist] [Rank 0] step=1830, skipped=3, lr=[7.053333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:22:41,192] [INFO] [timer.py:197:stop] 0/3660, RunningAvgSamplesPerSec=5.868122547434797, CurrSamplesPerSec=5.321938645914429, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1808/5000 [6:00:16<10:15:16, 11.57s/it][2022-12-20 16:22:52,141] [INFO] [timer.py:197:stop] 0/3662, RunningAvgSamplesPerSec=5.868145908661473, CurrSamplesPerSec=5.351630995168219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1809/5000 [6:00:27<10:05:15, 11.38s/it][2022-12-20 16:23:03,180] [INFO] [timer.py:197:stop] 0/3664, RunningAvgSamplesPerSec=5.868148154905698, CurrSamplesPerSec=5.329684530306949, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1810/5000 [6:00:38<9:59:36, 11.28s/it] [2022-12-20 16:23:14,210] [INFO] [timer.py:197:stop] 0/3666, RunningAvgSamplesPerSec=5.8681475644032295, CurrSamplesPerSec=5.334633614325137, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1811/5000 [6:00:49<9:55:28, 11.20s/it][2022-12-20 16:23:25,158] [INFO] [timer.py:197:stop] 0/3668, RunningAvgSamplesPerSec=5.868171638389008, CurrSamplesPerSec=5.346339181450542, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▌      | 1812/5000 [6:01:00<9:51:12, 11.13s/it][2022-12-20 16:23:36,165] [INFO] [timer.py:197:stop] 0/3670, RunningAvgSamplesPerSec=5.86817780599577, CurrSamplesPerSec=5.312544895366188, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1813/5000 [6:01:11<9:49:06, 11.09s/it][2022-12-20 16:23:47,196] [INFO] [timer.py:197:stop] 0/3672, RunningAvgSamplesPerSec=5.868177330254089, CurrSamplesPerSec=5.319049200096095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1814/5000 [6:01:22<9:47:58, 11.07s/it][2022-12-20 16:23:58,179] [INFO] [timer.py:197:stop] 0/3674, RunningAvgSamplesPerSec=5.868190599847353, CurrSamplesPerSec=5.333014189427325, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1815/5000 [6:01:33<9:46:21, 11.05s/it][2022-12-20 16:24:09,158] [INFO] [timer.py:197:stop] 0/3676, RunningAvgSamplesPerSec=5.86820545758403, CurrSamplesPerSec=5.308064903126967, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1816/5000 [6:01:44<9:45:06, 11.03s/it][2022-12-20 16:24:20,135] [INFO] [timer.py:197:stop] 0/3678, RunningAvgSamplesPerSec=5.868220665096575, CurrSamplesPerSec=5.328915972864234, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1817/5000 [6:01:55<9:44:09, 11.01s/it][2022-12-20 16:24:31,114] [INFO] [logging.py:68:log_dist] [Rank 0] step=1840, skipped=3, lr=[7.031111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:24:31,116] [INFO] [timer.py:197:stop] 0/3680, RunningAvgSamplesPerSec=5.868234756206267, CurrSamplesPerSec=5.33764657898069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1818/5000 [6:02:06<9:43:28, 11.00s/it][2022-12-20 16:24:42,103] [INFO] [timer.py:197:stop] 0/3682, RunningAvgSamplesPerSec=5.8682466593737255, CurrSamplesPerSec=5.321677201230961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1819/5000 [6:02:17<9:43:03, 11.00s/it][2022-12-20 16:24:53,135] [INFO] [timer.py:197:stop] 0/3684, RunningAvgSamplesPerSec=5.868245948061289, CurrSamplesPerSec=5.3332663650444605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1820/5000 [6:02:28<9:43:25, 11.01s/it][2022-12-20 16:25:04,128] [INFO] [timer.py:197:stop] 0/3686, RunningAvgSamplesPerSec=5.868256603460851, CurrSamplesPerSec=5.344668161250516, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1821/5000 [6:02:39<9:42:59, 11.00s/it][2022-12-20 16:25:15,101] [INFO] [timer.py:197:stop] 0/3688, RunningAvgSamplesPerSec=5.868272661115379, CurrSamplesPerSec=5.346490815049714, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1822/5000 [6:02:50<9:42:20, 10.99s/it][2022-12-20 16:25:26,085] [INFO] [timer.py:197:stop] 0/3690, RunningAvgSamplesPerSec=5.868285621717364, CurrSamplesPerSec=5.321523595996189, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1823/5000 [6:03:01<9:41:59, 10.99s/it][2022-12-20 16:25:37,094] [INFO] [timer.py:197:stop] 0/3692, RunningAvgSamplesPerSec=5.868294990879574, CurrSamplesPerSec=5.308947151348811, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1824/5000 [6:03:12<9:42:05, 11.00s/it][2022-12-20 16:25:48,087] [INFO] [timer.py:197:stop] 0/3694, RunningAvgSamplesPerSec=5.868305565250043, CurrSamplesPerSec=5.354673449952983, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 36%|███▋      | 1825/5000 [6:03:23<9:41:50, 11.00s/it]
                                                       
{'loss': 0.0003, 'learning_rate': 7.015555555555556e-06, 'epoch': 44.51}

 36%|███▋      | 1825/5000 [6:03:23<9:41:50, 11.00s/it][2022-12-20 16:25:59,075] [INFO] [timer.py:197:stop] 0/3696, RunningAvgSamplesPerSec=5.86831799793993, CurrSamplesPerSec=5.314430292496196, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1826/5000 [6:03:34<9:41:32, 10.99s/it][2022-12-20 16:26:10,088] [INFO] [timer.py:197:stop] 0/3698, RunningAvgSamplesPerSec=5.8683228582549685, CurrSamplesPerSec=5.294283772891498, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1827/5000 [6:03:45<9:41:40, 11.00s/it][2022-12-20 16:26:21,066] [INFO] [logging.py:68:log_dist] [Rank 0] step=1850, skipped=3, lr=[7.008888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:26:21,068] [INFO] [timer.py:197:stop] 0/3700, RunningAvgSamplesPerSec=5.868336823992637, CurrSamplesPerSec=5.329232298823237, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1828/5000 [6:03:56<9:41:11, 10.99s/it][2022-12-20 16:26:32,089] [INFO] [timer.py:197:stop] 0/3702, RunningAvgSamplesPerSec=5.868339276831172, CurrSamplesPerSec=5.3270109826711645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1829/5000 [6:04:07<9:41:26, 11.00s/it][2022-12-20 16:26:43,137] [INFO] [timer.py:197:stop] 0/3704, RunningAvgSamplesPerSec=5.8683339059503234, CurrSamplesPerSec=5.31480025068922, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1830/5000 [6:04:18<9:41:59, 11.02s/it][2022-12-20 16:26:54,190] [INFO] [timer.py:197:stop] 0/3706, RunningAvgSamplesPerSec=5.8683269539631775, CurrSamplesPerSec=5.306354781722552, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1831/5000 [6:04:29<9:42:23, 11.03s/it][2022-12-20 16:27:05,187] [INFO] [timer.py:197:stop] 0/3708, RunningAvgSamplesPerSec=5.86833620414536, CurrSamplesPerSec=5.307387983825914, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1832/5000 [6:04:40<9:41:45, 11.02s/it][2022-12-20 16:27:16,129] [INFO] [timer.py:197:stop] 0/3710, RunningAvgSamplesPerSec=5.8683618261204415, CurrSamplesPerSec=5.357746527674794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1833/5000 [6:04:51<9:40:21, 11.00s/it][2022-12-20 16:27:27,173] [INFO] [timer.py:197:stop] 0/3712, RunningAvgSamplesPerSec=5.868357495813902, CurrSamplesPerSec=5.3143345494768095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1834/5000 [6:05:02<9:40:57, 11.01s/it][2022-12-20 16:27:38,211] [INFO] [timer.py:197:stop] 0/3714, RunningAvgSamplesPerSec=5.8683550285875965, CurrSamplesPerSec=5.319246299738143, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1835/5000 [6:05:13<9:41:12, 11.02s/it][2022-12-20 16:27:49,192] [INFO] [timer.py:197:stop] 0/3716, RunningAvgSamplesPerSec=5.86836907975285, CurrSamplesPerSec=5.360113855825353, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1836/5000 [6:05:24<9:40:25, 11.01s/it][2022-12-20 16:28:00,148] [INFO] [timer.py:197:stop] 0/3718, RunningAvgSamplesPerSec=5.868390048636477, CurrSamplesPerSec=5.349072674212964, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1837/5000 [6:05:35<9:39:26, 10.99s/it][2022-12-20 16:28:11,101] [INFO] [logging.py:68:log_dist] [Rank 0] step=1860, skipped=3, lr=[6.986666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:28:11,103] [INFO] [timer.py:197:stop] 0/3720, RunningAvgSamplesPerSec=5.868411506969105, CurrSamplesPerSec=5.357620987604054, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1838/5000 [6:05:46<9:38:41, 10.98s/it][2022-12-20 16:28:22,109] [INFO] [timer.py:197:stop] 0/3722, RunningAvgSamplesPerSec=5.8684182052315625, CurrSamplesPerSec=5.3540307235296005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1839/5000 [6:05:57<9:38:53, 10.99s/it][2022-12-20 16:28:33,149] [INFO] [timer.py:197:stop] 0/3724, RunningAvgSamplesPerSec=5.868415356657042, CurrSamplesPerSec=5.323421495714795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1840/5000 [6:06:08<9:39:32, 11.00s/it][2022-12-20 16:28:44,160] [INFO] [timer.py:197:stop] 0/3726, RunningAvgSamplesPerSec=5.868425675215126, CurrSamplesPerSec=5.326175560257915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1841/5000 [6:06:19<9:39:27, 11.01s/it][2022-12-20 16:28:55,185] [INFO] [timer.py:197:stop] 0/3728, RunningAvgSamplesPerSec=5.86842671145372, CurrSamplesPerSec=5.325337441739497, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1842/5000 [6:06:30<9:39:34, 11.01s/it][2022-12-20 16:29:06,206] [INFO] [timer.py:197:stop] 0/3730, RunningAvgSamplesPerSec=5.868428850838629, CurrSamplesPerSec=5.323535091800804, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1843/5000 [6:06:41<9:39:33, 11.01s/it][2022-12-20 16:29:17,188] [INFO] [timer.py:197:stop] 0/3732, RunningAvgSamplesPerSec=5.868442745815802, CurrSamplesPerSec=5.361874445881324, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1844/5000 [6:06:52<9:38:51, 11.00s/it][2022-12-20 16:29:28,229] [INFO] [timer.py:197:stop] 0/3734, RunningAvgSamplesPerSec=5.868444475831424, CurrSamplesPerSec=5.317454811169712, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1845/5000 [6:07:03<9:39:13, 11.02s/it][2022-12-20 16:29:38,341] [INFO] [timer.py:197:stop] 0/3736, RunningAvgSamplesPerSec=5.868706050036927, CurrSamplesPerSec=5.343482538933694, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1846/5000 [6:07:18<10:42:43, 12.23s/it][2022-12-20 16:29:49,334] [INFO] [timer.py:197:stop] 0/3738, RunningAvgSamplesPerSec=5.868716611851059, CurrSamplesPerSec=5.344501095398371, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1847/5000 [6:07:29<10:23:20, 11.86s/it][2022-12-20 16:30:00,322] [INFO] [logging.py:68:log_dist] [Rank 0] step=1870, skipped=3, lr=[6.964444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:30:00,324] [INFO] [timer.py:197:stop] 0/3740, RunningAvgSamplesPerSec=5.86872799107172, CurrSamplesPerSec=5.362258323237146, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1848/5000 [6:07:40<10:09:15, 11.60s/it][2022-12-20 16:30:11,350] [INFO] [timer.py:197:stop] 0/3742, RunningAvgSamplesPerSec=5.868728892944803, CurrSamplesPerSec=5.322625614524423, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1849/5000 [6:07:51<10:00:23, 11.43s/it][2022-12-20 16:30:22,386] [INFO] [timer.py:197:stop] 0/3744, RunningAvgSamplesPerSec=5.868726741246671, CurrSamplesPerSec=5.331778026322872, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1850/5000 [6:08:02<9:52:53, 11.29s/it] 
                                                       
{'loss': 0.0003, 'learning_rate': 6.96e-06, 'epoch': 45.12}

 37%|███▋      | 1850/5000 [6:08:02<9:52:53, 11.29s/it][2022-12-20 16:30:33,352] [INFO] [timer.py:197:stop] 0/3746, RunningAvgSamplesPerSec=5.868744658743884, CurrSamplesPerSec=5.334493253512511, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1851/5000 [6:08:13<9:48:12, 11.21s/it][2022-12-20 16:30:44,342] [INFO] [timer.py:197:stop] 0/3748, RunningAvgSamplesPerSec=5.868755344175072, CurrSamplesPerSec=5.349467726523821, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1852/5000 [6:08:24<9:44:56, 11.15s/it][2022-12-20 16:30:55,385] [INFO] [timer.py:197:stop] 0/3750, RunningAvgSamplesPerSec=5.868751074232292, CurrSamplesPerSec=5.321030557789979, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1853/5000 [6:08:35<9:43:20, 11.12s/it][2022-12-20 16:31:06,471] [INFO] [timer.py:197:stop] 0/3752, RunningAvgSamplesPerSec=5.868734432402666, CurrSamplesPerSec=5.297403107625619, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1854/5000 [6:08:46<9:41:39, 11.09s/it][2022-12-20 16:31:17,470] [INFO] [timer.py:197:stop] 0/3754, RunningAvgSamplesPerSec=5.86874245227039, CurrSamplesPerSec=5.321080342695744, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1855/5000 [6:08:58<9:40:16, 11.07s/it][2022-12-20 16:31:28,485] [INFO] [timer.py:197:stop] 0/3756, RunningAvgSamplesPerSec=5.868746021327581, CurrSamplesPerSec=5.323925536132185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1856/5000 [6:09:09<9:39:34, 11.06s/it][2022-12-20 16:31:39,627] [INFO] [timer.py:197:stop] 0/3758, RunningAvgSamplesPerSec=5.868713093712055, CurrSamplesPerSec=5.232180599438258, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1857/5000 [6:09:20<9:40:03, 11.07s/it][2022-12-20 16:31:50,602] [INFO] [logging.py:68:log_dist] [Rank 0] step=1880, skipped=3, lr=[6.942222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:31:50,604] [INFO] [timer.py:197:stop] 0/3760, RunningAvgSamplesPerSec=5.868727513110509, CurrSamplesPerSec=5.342600897931986, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1858/5000 [6:09:31<9:38:54, 11.05s/it][2022-12-20 16:32:01,632] [INFO] [timer.py:197:stop] 0/3762, RunningAvgSamplesPerSec=5.868727241214993, CurrSamplesPerSec=5.328374391530989, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1859/5000 [6:09:42<9:38:05, 11.04s/it][2022-12-20 16:32:12,634] [INFO] [timer.py:197:stop] 0/3764, RunningAvgSamplesPerSec=5.868736391370496, CurrSamplesPerSec=5.345233283039553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1860/5000 [6:09:53<9:37:17, 11.03s/it][2022-12-20 16:32:23,667] [INFO] [timer.py:197:stop] 0/3766, RunningAvgSamplesPerSec=5.868735681107274, CurrSamplesPerSec=5.315751898546076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1861/5000 [6:10:04<9:36:25, 11.02s/it][2022-12-20 16:32:34,616] [INFO] [timer.py:197:stop] 0/3768, RunningAvgSamplesPerSec=5.868758489828963, CurrSamplesPerSec=5.348920680962366, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1862/5000 [6:10:15<9:35:27, 11.00s/it][2022-12-20 16:32:45,580] [INFO] [timer.py:197:stop] 0/3770, RunningAvgSamplesPerSec=5.868776839356115, CurrSamplesPerSec=5.351711015472656, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1863/5000 [6:10:26<9:35:09, 11.00s/it][2022-12-20 16:32:56,635] [INFO] [timer.py:197:stop] 0/3772, RunningAvgSamplesPerSec=5.868768565999091, CurrSamplesPerSec=5.298422368231982, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1864/5000 [6:10:37<9:35:35, 11.01s/it][2022-12-20 16:33:07,622] [INFO] [timer.py:197:stop] 0/3774, RunningAvgSamplesPerSec=5.8687804552120175, CurrSamplesPerSec=5.346630743009816, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1865/5000 [6:10:48<9:34:46, 11.00s/it][2022-12-20 16:33:18,617] [INFO] [timer.py:197:stop] 0/3776, RunningAvgSamplesPerSec=5.868789928065298, CurrSamplesPerSec=5.325936523777543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1866/5000 [6:10:59<9:34:16, 10.99s/it][2022-12-20 16:33:29,580] [INFO] [timer.py:197:stop] 0/3778, RunningAvgSamplesPerSec=5.868808120047668, CurrSamplesPerSec=5.34097769560886, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1867/5000 [6:11:10<9:34:33, 11.00s/it][2022-12-20 16:33:40,600] [INFO] [logging.py:68:log_dist] [Rank 0] step=1890, skipped=3, lr=[6.92e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:33:40,602] [INFO] [timer.py:197:stop] 0/3780, RunningAvgSamplesPerSec=5.868809453467568, CurrSamplesPerSec=5.3425175346367935, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1868/5000 [6:11:21<9:34:12, 11.00s/it][2022-12-20 16:33:51,599] [INFO] [timer.py:197:stop] 0/3782, RunningAvgSamplesPerSec=5.868818229839722, CurrSamplesPerSec=5.34016359590789, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1869/5000 [6:11:32<9:34:04, 11.00s/it][2022-12-20 16:34:02,628] [INFO] [timer.py:197:stop] 0/3784, RunningAvgSamplesPerSec=5.868817758535052, CurrSamplesPerSec=5.316844998430708, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1870/5000 [6:11:43<9:33:57, 11.00s/it][2022-12-20 16:34:13,633] [INFO] [timer.py:197:stop] 0/3786, RunningAvgSamplesPerSec=5.868823979332049, CurrSamplesPerSec=5.315866641334482, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1871/5000 [6:11:54<9:34:40, 11.02s/it][2022-12-20 16:34:24,602] [INFO] [timer.py:197:stop] 0/3788, RunningAvgSamplesPerSec=5.868845354753321, CurrSamplesPerSec=5.398444182432372, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1872/5000 [6:12:05<9:33:13, 11.00s/it][2022-12-20 16:34:35,609] [INFO] [timer.py:197:stop] 0/3790, RunningAvgSamplesPerSec=5.868851344861927, CurrSamplesPerSec=5.336599653739395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1873/5000 [6:12:16<9:33:08, 11.00s/it][2022-12-20 16:34:46,671] [INFO] [timer.py:197:stop] 0/3792, RunningAvgSamplesPerSec=5.868841521140663, CurrSamplesPerSec=5.283556990771648, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 37%|███▋      | 1874/5000 [6:12:27<9:33:43, 11.01s/it][2022-12-20 16:34:57,682] [INFO] [timer.py:197:stop] 0/3794, RunningAvgSamplesPerSec=5.8688459956298775, CurrSamplesPerSec=5.314820665126496, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1875/5000 [6:12:38<9:33:45, 11.02s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.904444444444444e-06, 'epoch': 45.72}

 38%|███▊      | 1875/5000 [6:12:38<9:33:45, 11.02s/it][2022-12-20 16:35:08,713] [INFO] [timer.py:197:stop] 0/3796, RunningAvgSamplesPerSec=5.86884505569269, CurrSamplesPerSec=5.311306221970929, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1876/5000 [6:12:49<9:33:36, 11.02s/it][2022-12-20 16:35:19,700] [INFO] [timer.py:197:stop] 0/3798, RunningAvgSamplesPerSec=5.868857147887066, CurrSamplesPerSec=5.339707460077929, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1877/5000 [6:13:00<9:33:14, 11.01s/it][2022-12-20 16:35:30,766] [INFO] [logging.py:68:log_dist] [Rank 0] step=1900, skipped=3, lr=[6.897777777777779e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:35:30,767] [INFO] [timer.py:197:stop] 0/3800, RunningAvgSamplesPerSec=5.868845747112066, CurrSamplesPerSec=5.282470881175296, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1878/5000 [6:13:11<9:34:03, 11.03s/it][2022-12-20 16:35:41,804] [INFO] [timer.py:197:stop] 0/3802, RunningAvgSamplesPerSec=5.868843133515691, CurrSamplesPerSec=5.319080819375827, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1879/5000 [6:13:22<9:33:42, 11.03s/it][2022-12-20 16:35:52,835] [INFO] [timer.py:197:stop] 0/3804, RunningAvgSamplesPerSec=5.868841949957179, CurrSamplesPerSec=5.3108411328482585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1880/5000 [6:13:33<9:33:14, 11.02s/it][2022-12-20 16:36:03,804] [INFO] [timer.py:197:stop] 0/3806, RunningAvgSamplesPerSec=5.868858912477242, CurrSamplesPerSec=5.348922812645288, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1881/5000 [6:13:44<9:31:54, 11.00s/it][2022-12-20 16:36:14,757] [INFO] [timer.py:197:stop] 0/3808, RunningAvgSamplesPerSec=5.868879766394498, CurrSamplesPerSec=5.345230728547926, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1882/5000 [6:13:55<9:32:02, 11.01s/it][2022-12-20 16:36:25,782] [INFO] [timer.py:197:stop] 0/3810, RunningAvgSamplesPerSec=5.868880605851189, CurrSamplesPerSec=5.34352189519481, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1883/5000 [6:14:06<9:32:01, 11.01s/it][2022-12-20 16:36:36,783] [INFO] [timer.py:197:stop] 0/3812, RunningAvgSamplesPerSec=5.86888787766575, CurrSamplesPerSec=5.358793423001814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1884/5000 [6:14:17<9:31:01, 11.00s/it][2022-12-20 16:36:47,727] [INFO] [timer.py:197:stop] 0/3814, RunningAvgSamplesPerSec=5.868911516982396, CurrSamplesPerSec=5.372213307296083, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1885/5000 [6:14:28<9:29:52, 10.98s/it][2022-12-20 16:36:58,664] [INFO] [timer.py:197:stop] 0/3816, RunningAvgSamplesPerSec=5.86893729650693, CurrSamplesPerSec=5.369476095965548, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1886/5000 [6:14:39<9:29:01, 10.96s/it][2022-12-20 16:37:08,748] [INFO] [timer.py:197:stop] 0/3818, RunningAvgSamplesPerSec=5.8692004493378125, CurrSamplesPerSec=6.248256142802676, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 16:37:19,733] [INFO] [logging.py:68:log_dist] [Rank 0] step=1910, skipped=3, lr=[6.875555555555557e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:37:19,734] [INFO] [timer.py:197:stop] 0/3820, RunningAvgSamplesPerSec=5.869211360605502, CurrSamplesPerSec=5.316808140312871, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1887/5000 [6:14:55<10:49:43, 12.52s/it][2022-12-20 16:37:30,715] [INFO] [timer.py:197:stop] 0/3822, RunningAvgSamplesPerSec=5.8692244754562966, CurrSamplesPerSec=5.361287168495422, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1888/5000 [6:15:06<10:25:31, 12.06s/it][2022-12-20 16:37:41,727] [INFO] [timer.py:197:stop] 0/3824, RunningAvgSamplesPerSec=5.869228957885405, CurrSamplesPerSec=5.316804138605116, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1889/5000 [6:15:17<10:09:00, 11.75s/it][2022-12-20 16:37:52,701] [INFO] [timer.py:197:stop] 0/3826, RunningAvgSamplesPerSec=5.869243675413664, CurrSamplesPerSec=5.348302990836642, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1890/5000 [6:15:28<9:56:49, 11.51s/it] [2022-12-20 16:38:03,784] [INFO] [timer.py:197:stop] 0/3828, RunningAvgSamplesPerSec=5.869227979587994, CurrSamplesPerSec=5.286715545388966, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1891/5000 [6:15:39<9:49:55, 11.38s/it][2022-12-20 16:38:14,821] [INFO] [timer.py:197:stop] 0/3830, RunningAvgSamplesPerSec=5.869225616368315, CurrSamplesPerSec=5.3182169050599795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1892/5000 [6:15:50<9:44:19, 11.28s/it][2022-12-20 16:38:25,878] [INFO] [timer.py:197:stop] 0/3832, RunningAvgSamplesPerSec=5.86921717053297, CurrSamplesPerSec=5.318021988820157, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1893/5000 [6:16:01<9:40:40, 11.21s/it][2022-12-20 16:38:36,883] [INFO] [timer.py:197:stop] 0/3834, RunningAvgSamplesPerSec=5.869223571325087, CurrSamplesPerSec=5.3388340722982335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1894/5000 [6:16:12<9:37:14, 11.15s/it][2022-12-20 16:38:47,958] [INFO] [timer.py:197:stop] 0/3836, RunningAvgSamplesPerSec=5.869210283027008, CurrSamplesPerSec=5.269363905321959, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1895/5000 [6:16:23<9:35:53, 11.13s/it][2022-12-20 16:38:58,908] [INFO] [timer.py:197:stop] 0/3838, RunningAvgSamplesPerSec=5.869232230158236, CurrSamplesPerSec=5.350949959199412, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1896/5000 [6:16:34<9:32:55, 11.07s/it][2022-12-20 16:39:09,961] [INFO] [logging.py:68:log_dist] [Rank 0] step=1920, skipped=3, lr=[6.853333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:39:09,963] [INFO] [timer.py:197:stop] 0/3840, RunningAvgSamplesPerSec=5.869224439634074, CurrSamplesPerSec=5.306866296574155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1897/5000 [6:16:45<9:32:26, 11.07s/it][2022-12-20 16:39:20,990] [INFO] [timer.py:197:stop] 0/3842, RunningAvgSamplesPerSec=5.869224567054846, CurrSamplesPerSec=5.333898604217269, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1898/5000 [6:16:56<9:31:36, 11.06s/it][2022-12-20 16:39:31,991] [INFO] [timer.py:197:stop] 0/3844, RunningAvgSamplesPerSec=5.869231945721757, CurrSamplesPerSec=5.32162318519249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1899/5000 [6:17:07<9:30:34, 11.04s/it][2022-12-20 16:39:42,964] [INFO] [timer.py:197:stop] 0/3846, RunningAvgSamplesPerSec=5.869247350035886, CurrSamplesPerSec=5.34918331708863, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1900/5000 [6:17:18<9:29:21, 11.02s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.846666666666667e-06, 'epoch': 46.34}

 38%|███▊      | 1900/5000 [6:17:18<9:29:21, 11.02s/it][2022-12-20 16:39:53,976] [INFO] [timer.py:197:stop] 0/3848, RunningAvgSamplesPerSec=5.869252167410342, CurrSamplesPerSec=5.308627769663831, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1901/5000 [6:17:29<9:29:03, 11.02s/it][2022-12-20 16:40:04,999] [INFO] [timer.py:197:stop] 0/3850, RunningAvgSamplesPerSec=5.869253645555402, CurrSamplesPerSec=5.323418539747536, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1902/5000 [6:17:40<9:28:57, 11.02s/it][2022-12-20 16:40:15,973] [INFO] [timer.py:197:stop] 0/3852, RunningAvgSamplesPerSec=5.869268764376935, CurrSamplesPerSec=5.359081209063046, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1903/5000 [6:17:51<9:28:03, 11.01s/it][2022-12-20 16:40:26,946] [INFO] [timer.py:197:stop] 0/3854, RunningAvgSamplesPerSec=5.869283832861956, CurrSamplesPerSec=5.359023221429588, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1904/5000 [6:18:02<9:27:22, 11.00s/it][2022-12-20 16:40:37,969] [INFO] [timer.py:197:stop] 0/3856, RunningAvgSamplesPerSec=5.869285350627309, CurrSamplesPerSec=5.336374956861386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1905/5000 [6:18:13<9:27:37, 11.00s/it][2022-12-20 16:40:48,982] [INFO] [timer.py:197:stop] 0/3858, RunningAvgSamplesPerSec=5.869289213648897, CurrSamplesPerSec=5.308052097779914, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1906/5000 [6:18:24<9:27:34, 11.01s/it][2022-12-20 16:40:59,941] [INFO] [logging.py:68:log_dist] [Rank 0] step=1930, skipped=3, lr=[6.831111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:40:59,943] [INFO] [timer.py:197:stop] 0/3860, RunningAvgSamplesPerSec=5.869307706769644, CurrSamplesPerSec=5.353391567774957, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1907/5000 [6:18:35<9:26:41, 10.99s/it][2022-12-20 16:41:10,899] [INFO] [timer.py:197:stop] 0/3862, RunningAvgSamplesPerSec=5.869327569959282, CurrSamplesPerSec=5.361225064259297, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1908/5000 [6:18:46<9:25:55, 10.98s/it][2022-12-20 16:41:21,895] [INFO] [timer.py:197:stop] 0/3864, RunningAvgSamplesPerSec=5.869336043827087, CurrSamplesPerSec=5.313138578555798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1909/5000 [6:18:57<9:25:58, 10.99s/it][2022-12-20 16:41:32,914] [INFO] [timer.py:197:stop] 0/3866, RunningAvgSamplesPerSec=5.869342909431282, CurrSamplesPerSec=5.322120976054332, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1910/5000 [6:19:08<9:26:18, 11.00s/it][2022-12-20 16:41:43,892] [INFO] [timer.py:197:stop] 0/3868, RunningAvgSamplesPerSec=5.8693570598635665, CurrSamplesPerSec=5.341944907623823, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1911/5000 [6:19:19<9:25:49, 10.99s/it][2022-12-20 16:41:54,890] [INFO] [timer.py:197:stop] 0/3870, RunningAvgSamplesPerSec=5.8693651980584844, CurrSamplesPerSec=5.325207077368411, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1912/5000 [6:19:30<9:25:45, 10.99s/it][2022-12-20 16:42:05,881] [INFO] [timer.py:197:stop] 0/3872, RunningAvgSamplesPerSec=5.86937528936078, CurrSamplesPerSec=5.364937139995612, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1913/5000 [6:19:41<9:25:33, 10.99s/it][2022-12-20 16:42:16,906] [INFO] [timer.py:197:stop] 0/3874, RunningAvgSamplesPerSec=5.869375984128186, CurrSamplesPerSec=5.329702307936765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1914/5000 [6:19:52<9:25:52, 11.00s/it][2022-12-20 16:42:27,887] [INFO] [timer.py:197:stop] 0/3876, RunningAvgSamplesPerSec=5.869388822495246, CurrSamplesPerSec=5.339384366756276, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1915/5000 [6:20:03<9:25:21, 11.00s/it][2022-12-20 16:42:38,829] [INFO] [timer.py:197:stop] 0/3878, RunningAvgSamplesPerSec=5.869412413194206, CurrSamplesPerSec=5.3402938436812795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1916/5000 [6:20:14<9:24:21, 10.98s/it][2022-12-20 16:42:49,879] [INFO] [logging.py:68:log_dist] [Rank 0] step=1940, skipped=3, lr=[6.80888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:42:49,881] [INFO] [timer.py:197:stop] 0/3880, RunningAvgSamplesPerSec=5.869405436560416, CurrSamplesPerSec=5.298367986609769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1917/5000 [6:20:25<9:25:17, 11.00s/it][2022-12-20 16:43:00,901] [INFO] [timer.py:197:stop] 0/3882, RunningAvgSamplesPerSec=5.8694072961784345, CurrSamplesPerSec=5.335487387651544, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1918/5000 [6:20:36<9:25:23, 11.01s/it][2022-12-20 16:43:11,947] [INFO] [timer.py:197:stop] 0/3884, RunningAvgSamplesPerSec=5.869402042839468, CurrSamplesPerSec=5.307456402552905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1919/5000 [6:20:47<9:25:48, 11.02s/it][2022-12-20 16:43:22,963] [INFO] [timer.py:197:stop] 0/3886, RunningAvgSamplesPerSec=5.869405125708295, CurrSamplesPerSec=5.307123769780808, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1920/5000 [6:20:58<9:25:35, 11.02s/it][2022-12-20 16:43:34,045] [INFO] [timer.py:197:stop] 0/3888, RunningAvgSamplesPerSec=5.869391301553758, CurrSamplesPerSec=5.276328210939081, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1921/5000 [6:21:09<9:26:23, 11.04s/it][2022-12-20 16:43:45,150] [INFO] [timer.py:197:stop] 0/3890, RunningAvgSamplesPerSec=5.869369935923021, CurrSamplesPerSec=5.2670712880852255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1922/5000 [6:21:20<9:27:14, 11.06s/it][2022-12-20 16:43:56,273] [INFO] [timer.py:197:stop] 0/3892, RunningAvgSamplesPerSec=5.869344162309848, CurrSamplesPerSec=5.249545282025271, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1923/5000 [6:21:31<9:28:04, 11.08s/it][2022-12-20 16:44:07,263] [INFO] [timer.py:197:stop] 0/3894, RunningAvgSamplesPerSec=5.869354858142882, CurrSamplesPerSec=5.352553827934376, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1924/5000 [6:21:42<9:26:32, 11.05s/it][2022-12-20 16:44:18,300] [INFO] [timer.py:197:stop] 0/3896, RunningAvgSamplesPerSec=5.869352430820795, CurrSamplesPerSec=5.3142569054734805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 38%|███▊      | 1925/5000 [6:21:53<9:26:08, 11.05s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.7911111111111115e-06, 'epoch': 46.94}

 38%|███▊      | 1925/5000 [6:21:53<9:26:08, 11.05s/it][2022-12-20 16:44:29,342] [INFO] [timer.py:197:stop] 0/3898, RunningAvgSamplesPerSec=5.86934956407227, CurrSamplesPerSec=5.329271445449385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1926/5000 [6:22:04<9:25:53, 11.05s/it][2022-12-20 16:44:40,396] [INFO] [logging.py:68:log_dist] [Rank 0] step=1950, skipped=3, lr=[6.786666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:44:40,397] [INFO] [timer.py:197:stop] 0/3900, RunningAvgSamplesPerSec=5.8693419947365815, CurrSamplesPerSec=5.282200618872502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1927/5000 [6:22:15<9:25:51, 11.05s/it][2022-12-20 16:44:50,510] [INFO] [timer.py:197:stop] 0/3902, RunningAvgSamplesPerSec=5.869591892280097, CurrSamplesPerSec=5.318891952461886, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1928/5000 [6:22:31<10:27:46, 12.26s/it][2022-12-20 16:45:01,546] [INFO] [timer.py:197:stop] 0/3904, RunningAvgSamplesPerSec=5.869593820351264, CurrSamplesPerSec=5.338881430115715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1929/5000 [6:22:42<10:08:25, 11.89s/it][2022-12-20 16:45:12,603] [INFO] [timer.py:197:stop] 0/3906, RunningAvgSamplesPerSec=5.869586007882187, CurrSamplesPerSec=5.30195873203379, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1930/5000 [6:22:53<9:54:47, 11.62s/it] [2022-12-20 16:45:23,582] [INFO] [timer.py:197:stop] 0/3908, RunningAvgSamplesPerSec=5.86959881910179, CurrSamplesPerSec=5.329788234819466, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1931/5000 [6:23:04<9:45:10, 11.44s/it][2022-12-20 16:45:34,643] [INFO] [timer.py:197:stop] 0/3910, RunningAvgSamplesPerSec=5.869589249658905, CurrSamplesPerSec=5.285602117931175, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1932/5000 [6:23:15<9:39:38, 11.34s/it][2022-12-20 16:45:45,642] [INFO] [timer.py:197:stop] 0/3912, RunningAvgSamplesPerSec=5.869596773633829, CurrSamplesPerSec=5.367759885341876, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1933/5000 [6:23:26<9:34:01, 11.23s/it][2022-12-20 16:45:56,700] [INFO] [timer.py:197:stop] 0/3914, RunningAvgSamplesPerSec=5.869587751762808, CurrSamplesPerSec=5.300550402484532, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1934/5000 [6:23:37<9:31:25, 11.18s/it][2022-12-20 16:46:07,761] [INFO] [timer.py:197:stop] 0/3916, RunningAvgSamplesPerSec=5.869577869459803, CurrSamplesPerSec=5.309664167748668, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1935/5000 [6:23:48<9:29:19, 11.14s/it][2022-12-20 16:46:18,780] [INFO] [timer.py:197:stop] 0/3918, RunningAvgSamplesPerSec=5.869579894212139, CurrSamplesPerSec=5.344679654062521, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1936/5000 [6:23:59<9:27:11, 11.11s/it][2022-12-20 16:46:29,819] [INFO] [logging.py:68:log_dist] [Rank 0] step=1960, skipped=3, lr=[6.764444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:46:29,821] [INFO] [timer.py:197:stop] 0/3920, RunningAvgSamplesPerSec=5.8695752995091395, CurrSamplesPerSec=5.323166238926243, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▊      | 1937/5000 [6:24:10<9:25:55, 11.09s/it][2022-12-20 16:46:40,871] [INFO] [timer.py:197:stop] 0/3922, RunningAvgSamplesPerSec=5.869568734523539, CurrSamplesPerSec=5.312787568104147, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1938/5000 [6:24:21<9:25:21, 11.08s/it][2022-12-20 16:46:51,886] [INFO] [timer.py:197:stop] 0/3924, RunningAvgSamplesPerSec=5.869571431539732, CurrSamplesPerSec=5.3528223714273935, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1939/5000 [6:24:32<9:23:46, 11.05s/it][2022-12-20 16:47:02,908] [INFO] [timer.py:197:stop] 0/3926, RunningAvgSamplesPerSec=5.869572475989458, CurrSamplesPerSec=5.3214356145743675, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1940/5000 [6:24:43<9:23:15, 11.04s/it][2022-12-20 16:47:13,934] [INFO] [timer.py:197:stop] 0/3928, RunningAvgSamplesPerSec=5.86957576868585, CurrSamplesPerSec=5.334026426729179, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1941/5000 [6:24:54<9:22:10, 11.03s/it][2022-12-20 16:47:24,944] [INFO] [timer.py:197:stop] 0/3930, RunningAvgSamplesPerSec=5.869580090139969, CurrSamplesPerSec=5.302932185522639, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1942/5000 [6:25:05<9:21:50, 11.02s/it][2022-12-20 16:47:35,916] [INFO] [timer.py:197:stop] 0/3932, RunningAvgSamplesPerSec=5.86959479072205, CurrSamplesPerSec=5.342879077955207, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1943/5000 [6:25:16<9:20:48, 11.01s/it][2022-12-20 16:47:46,911] [INFO] [timer.py:197:stop] 0/3934, RunningAvgSamplesPerSec=5.869603626291639, CurrSamplesPerSec=5.319073230714407, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1944/5000 [6:25:27<9:20:58, 11.01s/it][2022-12-20 16:47:58,016] [INFO] [timer.py:197:stop] 0/3936, RunningAvgSamplesPerSec=5.869582091734974, CurrSamplesPerSec=5.253546135170918, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1945/5000 [6:25:38<9:22:14, 11.04s/it][2022-12-20 16:48:09,019] [INFO] [timer.py:197:stop] 0/3938, RunningAvgSamplesPerSec=5.869592221354327, CurrSamplesPerSec=5.349160505947804, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1946/5000 [6:25:49<9:21:15, 11.03s/it][2022-12-20 16:48:20,121] [INFO] [logging.py:68:log_dist] [Rank 0] step=1970, skipped=3, lr=[6.742222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:48:20,122] [INFO] [timer.py:197:stop] 0/3940, RunningAvgSamplesPerSec=5.8695713544378565, CurrSamplesPerSec=5.24730946709312, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1947/5000 [6:26:00<9:22:19, 11.05s/it][2022-12-20 16:48:31,168] [INFO] [timer.py:197:stop] 0/3942, RunningAvgSamplesPerSec=5.869566223596272, CurrSamplesPerSec=5.301972974096133, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1948/5000 [6:26:11<9:22:29, 11.06s/it][2022-12-20 16:48:42,202] [INFO] [timer.py:197:stop] 0/3944, RunningAvgSamplesPerSec=5.869568570836033, CurrSamplesPerSec=5.336716783693784, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1949/5000 [6:26:22<9:22:48, 11.07s/it][2022-12-20 16:48:53,381] [INFO] [timer.py:197:stop] 0/3946, RunningAvgSamplesPerSec=5.869546987859969, CurrSamplesPerSec=5.31634482470698, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1950/5000 [6:26:33<9:22:39, 11.07s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.735555555555556e-06, 'epoch': 47.55}

 39%|███▉      | 1950/5000 [6:26:33<9:22:39, 11.07s/it][2022-12-20 16:49:04,422] [INFO] [timer.py:197:stop] 0/3948, RunningAvgSamplesPerSec=5.869543442685055, CurrSamplesPerSec=5.287364914245699, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1951/5000 [6:26:44<9:22:15, 11.06s/it][2022-12-20 16:49:15,432] [INFO] [timer.py:197:stop] 0/3950, RunningAvgSamplesPerSec=5.869552740584047, CurrSamplesPerSec=5.325567338062437, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1952/5000 [6:26:55<9:21:17, 11.05s/it][2022-12-20 16:49:26,460] [INFO] [timer.py:197:stop] 0/3952, RunningAvgSamplesPerSec=5.86955243940339, CurrSamplesPerSec=5.311936418424618, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1953/5000 [6:27:06<9:20:49, 11.04s/it][2022-12-20 16:49:37,442] [INFO] [timer.py:197:stop] 0/3954, RunningAvgSamplesPerSec=5.869564487154895, CurrSamplesPerSec=5.35502894884612, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1954/5000 [6:27:17<9:19:43, 11.03s/it][2022-12-20 16:49:48,450] [INFO] [timer.py:197:stop] 0/3956, RunningAvgSamplesPerSec=5.869574362158543, CurrSamplesPerSec=5.334361168436438, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1955/5000 [6:27:28<9:18:41, 11.01s/it][2022-12-20 16:49:59,405] [INFO] [timer.py:197:stop] 0/3958, RunningAvgSamplesPerSec=5.869594160762788, CurrSamplesPerSec=5.347177959341796, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1956/5000 [6:27:39<9:18:24, 11.01s/it][2022-12-20 16:50:10,422] [INFO] [logging.py:68:log_dist] [Rank 0] step=1980, skipped=3, lr=[6.720000000000001e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:50:10,423] [INFO] [timer.py:197:stop] 0/3960, RunningAvgSamplesPerSec=5.869596322675665, CurrSamplesPerSec=5.331401465337975, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1957/5000 [6:27:50<9:18:40, 11.02s/it][2022-12-20 16:50:21,433] [INFO] [timer.py:197:stop] 0/3962, RunningAvgSamplesPerSec=5.869601262165809, CurrSamplesPerSec=5.356232526202764, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1958/5000 [6:28:01<9:18:11, 11.01s/it][2022-12-20 16:50:32,488] [INFO] [timer.py:197:stop] 0/3964, RunningAvgSamplesPerSec=5.869593648334074, CurrSamplesPerSec=5.304033852812347, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1959/5000 [6:28:13<9:18:49, 11.03s/it][2022-12-20 16:50:43,505] [INFO] [timer.py:197:stop] 0/3966, RunningAvgSamplesPerSec=5.86959648903757, CurrSamplesPerSec=5.344486836758444, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1960/5000 [6:28:24<9:18:16, 11.02s/it][2022-12-20 16:50:54,555] [INFO] [timer.py:197:stop] 0/3968, RunningAvgSamplesPerSec=5.869590105769698, CurrSamplesPerSec=5.302128175647129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1961/5000 [6:28:35<9:18:03, 11.02s/it][2022-12-20 16:51:05,556] [INFO] [timer.py:197:stop] 0/3970, RunningAvgSamplesPerSec=5.869601622349099, CurrSamplesPerSec=5.328819495630727, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1962/5000 [6:28:46<9:18:24, 11.03s/it][2022-12-20 16:51:16,577] [INFO] [timer.py:197:stop] 0/3972, RunningAvgSamplesPerSec=5.869603423984077, CurrSamplesPerSec=5.34412465025393, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1963/5000 [6:28:57<9:17:25, 11.01s/it][2022-12-20 16:51:27,561] [INFO] [timer.py:197:stop] 0/3974, RunningAvgSamplesPerSec=5.869615143522958, CurrSamplesPerSec=5.336657157114371, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1964/5000 [6:29:08<9:18:03, 11.03s/it][2022-12-20 16:51:38,615] [INFO] [timer.py:197:stop] 0/3976, RunningAvgSamplesPerSec=5.869607727422724, CurrSamplesPerSec=5.347221204644711, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1965/5000 [6:29:19<9:17:27, 11.02s/it][2022-12-20 16:51:49,620] [INFO] [timer.py:197:stop] 0/3978, RunningAvgSamplesPerSec=5.869613895587286, CurrSamplesPerSec=5.344482793277385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1966/5000 [6:29:30<9:17:12, 11.02s/it][2022-12-20 16:52:00,647] [INFO] [logging.py:68:log_dist] [Rank 0] step=1990, skipped=3, lr=[6.6977777777777785e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:52:00,649] [INFO] [timer.py:197:stop] 0/3980, RunningAvgSamplesPerSec=5.869613029615043, CurrSamplesPerSec=5.332084736214061, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1967/5000 [6:29:41<9:17:16, 11.02s/it][2022-12-20 16:52:11,722] [INFO] [timer.py:197:stop] 0/3982, RunningAvgSamplesPerSec=5.869600435932706, CurrSamplesPerSec=5.299258103151714, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1968/5000 [6:29:52<9:17:30, 11.03s/it][2022-12-20 16:52:21,935] [INFO] [timer.py:197:stop] 0/3984, RunningAvgSamplesPerSec=5.869817961050262, CurrSamplesPerSec=6.14320397814663, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 16:52:32,952] [INFO] [timer.py:197:stop] 0/3986, RunningAvgSamplesPerSec=5.869819792663639, CurrSamplesPerSec=5.305473814264637, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1969/5000 [6:30:08<10:36:50, 12.61s/it][2022-12-20 16:52:44,007] [INFO] [timer.py:197:stop] 0/3988, RunningAvgSamplesPerSec=5.869811964981253, CurrSamplesPerSec=5.30674480811317, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1970/5000 [6:30:19<10:13:08, 12.14s/it][2022-12-20 16:52:55,012] [INFO] [timer.py:197:stop] 0/3990, RunningAvgSamplesPerSec=5.869817733474977, CurrSamplesPerSec=5.32188019317856, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1971/5000 [6:30:30<9:55:43, 11.80s/it] [2022-12-20 16:53:06,047] [INFO] [timer.py:197:stop] 0/3992, RunningAvgSamplesPerSec=5.8698159209907566, CurrSamplesPerSec=5.302158965698852, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1972/5000 [6:30:41<9:43:55, 11.57s/it][2022-12-20 16:53:17,069] [INFO] [timer.py:197:stop] 0/3994, RunningAvgSamplesPerSec=5.869817049151985, CurrSamplesPerSec=5.314862757244781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1973/5000 [6:30:52<9:35:26, 11.41s/it][2022-12-20 16:53:28,132] [INFO] [timer.py:197:stop] 0/3996, RunningAvgSamplesPerSec=5.869807233205918, CurrSamplesPerSec=5.306065708248419, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 39%|███▉      | 1974/5000 [6:31:03<9:30:03, 11.30s/it][2022-12-20 16:53:39,525] [INFO] [timer.py:197:stop] 0/3998, RunningAvgSamplesPerSec=5.869708698504063, CurrSamplesPerSec=5.027969838713019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1975/5000 [6:31:15<9:31:14, 11.33s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.677777777777779e-06, 'epoch': 48.17}

 40%|███▉      | 1975/5000 [6:31:15<9:31:14, 11.33s/it][2022-12-20 16:53:50,575] [INFO] [logging.py:68:log_dist] [Rank 0] step=2000, skipped=3, lr=[6.675555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:53:50,577] [INFO] [timer.py:197:stop] 0/4000, RunningAvgSamplesPerSec=5.869712959722663, CurrSamplesPerSec=5.320165799181944, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1976/5000 [6:31:26<9:26:49, 11.25s/it][2022-12-20 16:54:01,622] [INFO] [timer.py:197:stop] 0/4002, RunningAvgSamplesPerSec=5.869707890884144, CurrSamplesPerSec=5.313586189414462, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1977/5000 [6:31:37<9:23:35, 11.19s/it][2022-12-20 16:54:12,689] [INFO] [timer.py:197:stop] 0/4004, RunningAvgSamplesPerSec=5.869697027754707, CurrSamplesPerSec=5.288194455662534, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1978/5000 [6:31:48<9:21:37, 11.15s/it][2022-12-20 16:54:23,765] [INFO] [timer.py:197:stop] 0/4006, RunningAvgSamplesPerSec=5.869687648830099, CurrSamplesPerSec=5.3106139770350635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1979/5000 [6:31:59<9:20:17, 11.13s/it][2022-12-20 16:54:34,803] [INFO] [timer.py:197:stop] 0/4008, RunningAvgSamplesPerSec=5.869689189018972, CurrSamplesPerSec=5.320263861331488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1980/5000 [6:32:10<9:18:44, 11.10s/it][2022-12-20 16:54:45,962] [INFO] [timer.py:197:stop] 0/4010, RunningAvgSamplesPerSec=5.869654010771513, CurrSamplesPerSec=5.213757295916811, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1981/5000 [6:32:21<9:19:27, 11.12s/it][2022-12-20 16:54:56,988] [INFO] [timer.py:197:stop] 0/4012, RunningAvgSamplesPerSec=5.8696706573841855, CurrSamplesPerSec=5.345509608299978, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1982/5000 [6:32:32<9:17:51, 11.09s/it][2022-12-20 16:55:08,008] [INFO] [timer.py:197:stop] 0/4014, RunningAvgSamplesPerSec=5.869672388766631, CurrSamplesPerSec=5.30914980331051, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1983/5000 [6:32:43<9:16:36, 11.07s/it][2022-12-20 16:55:19,149] [INFO] [timer.py:197:stop] 0/4016, RunningAvgSamplesPerSec=5.869641747035894, CurrSamplesPerSec=5.216913627561566, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1984/5000 [6:32:54<9:17:30, 11.09s/it][2022-12-20 16:55:30,160] [INFO] [timer.py:197:stop] 0/4018, RunningAvgSamplesPerSec=5.869645889132529, CurrSamplesPerSec=5.339104426646775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1985/5000 [6:33:05<9:16:07, 11.07s/it][2022-12-20 16:55:41,210] [INFO] [logging.py:68:log_dist] [Rank 0] step=2010, skipped=3, lr=[6.653333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:55:41,211] [INFO] [timer.py:197:stop] 0/4020, RunningAvgSamplesPerSec=5.869639377105949, CurrSamplesPerSec=5.321645973074997, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1986/5000 [6:33:16<9:15:41, 11.06s/it][2022-12-20 16:55:52,276] [INFO] [timer.py:197:stop] 0/4022, RunningAvgSamplesPerSec=5.86962927610254, CurrSamplesPerSec=5.249277762322744, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1987/5000 [6:33:27<9:15:33, 11.06s/it][2022-12-20 16:56:03,323] [INFO] [timer.py:197:stop] 0/4024, RunningAvgSamplesPerSec=5.869632974194579, CurrSamplesPerSec=5.303785062744144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1988/5000 [6:33:38<9:15:07, 11.06s/it][2022-12-20 16:56:14,306] [INFO] [timer.py:197:stop] 0/4026, RunningAvgSamplesPerSec=5.869644635905818, CurrSamplesPerSec=5.339517762810007, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1989/5000 [6:33:49<9:13:48, 11.04s/it][2022-12-20 16:56:25,471] [INFO] [timer.py:197:stop] 0/4028, RunningAvgSamplesPerSec=5.8696079325847865, CurrSamplesPerSec=5.183891165950706, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1990/5000 [6:34:01<9:15:34, 11.07s/it][2022-12-20 16:56:36,505] [INFO] [timer.py:197:stop] 0/4030, RunningAvgSamplesPerSec=5.869617051723272, CurrSamplesPerSec=5.3520750847196945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1991/5000 [6:34:12<9:14:46, 11.06s/it][2022-12-20 16:56:47,521] [INFO] [timer.py:197:stop] 0/4032, RunningAvgSamplesPerSec=5.869619648895697, CurrSamplesPerSec=5.3190930455981045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1992/5000 [6:34:23<9:13:53, 11.05s/it][2022-12-20 16:56:58,802] [INFO] [timer.py:197:stop] 0/4034, RunningAvgSamplesPerSec=5.869551547210776, CurrSamplesPerSec=5.097142153163929, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1993/5000 [6:34:34<9:17:13, 11.12s/it][2022-12-20 16:57:09,875] [INFO] [timer.py:197:stop] 0/4036, RunningAvgSamplesPerSec=5.869554301741147, CurrSamplesPerSec=5.328986428840508, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1994/5000 [6:34:45<9:16:20, 11.10s/it][2022-12-20 16:57:20,889] [INFO] [timer.py:197:stop] 0/4038, RunningAvgSamplesPerSec=5.869557895466522, CurrSamplesPerSec=5.345931815233471, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1995/5000 [6:34:56<9:14:46, 11.08s/it][2022-12-20 16:57:32,254] [INFO] [logging.py:68:log_dist] [Rank 0] step=2020, skipped=3, lr=[6.631111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 16:57:32,255] [INFO] [timer.py:197:stop] 0/4040, RunningAvgSamplesPerSec=5.86946724439319, CurrSamplesPerSec=5.045618774681076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1996/5000 [6:35:07<9:18:56, 11.16s/it][2022-12-20 16:57:43,330] [INFO] [timer.py:197:stop] 0/4042, RunningAvgSamplesPerSec=5.869454816246396, CurrSamplesPerSec=5.312724479471249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1997/5000 [6:35:18<9:17:25, 11.14s/it][2022-12-20 16:57:54,315] [INFO] [timer.py:197:stop] 0/4044, RunningAvgSamplesPerSec=5.869466283782735, CurrSamplesPerSec=5.317001703790771, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1998/5000 [6:35:29<9:14:56, 11.09s/it][2022-12-20 16:58:05,347] [INFO] [timer.py:197:stop] 0/4046, RunningAvgSamplesPerSec=5.869465102326746, CurrSamplesPerSec=5.327271894477746, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|███▉      | 1999/5000 [6:35:40<9:13:51, 11.07s/it][2022-12-20 16:58:16,506] [INFO] [timer.py:197:stop] 0/4048, RunningAvgSamplesPerSec=5.869473178920971, CurrSamplesPerSec=5.321830393304674, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2000/5000 [6:35:52<9:14:57, 11.10s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.6222222222222236e-06, 'epoch': 48.77}

 40%|████      | 2000/5000 [6:35:52<9:14:57, 11.10s/it][INFO|trainer.py:2955] 2022-12-20 16:58:16,513 >> ***** Running Evaluation *****
[INFO|trainer.py:2957] 2022-12-20 16:58:16,513 >>   Num examples = 1325
[INFO|trainer.py:2960] 2022-12-20 16:58:16,513 >>   Batch size = 8


  0%|          | 0/166 [00:00<?, ?it/s]

  1%|          | 2/166 [00:06<08:16,  3.03s/it]

  2%|▏         | 3/166 [00:11<11:22,  4.19s/it]

  2%|▏         | 4/166 [00:18<13:21,  4.95s/it]

  3%|▎         | 5/166 [00:25<15:22,  5.73s/it]

  4%|▎         | 6/166 [00:32<16:38,  6.24s/it]

  4%|▍         | 7/166 [00:38<16:33,  6.25s/it]

  5%|▍         | 8/166 [00:44<16:05,  6.11s/it]

  5%|▌         | 9/166 [00:50<16:10,  6.18s/it]

  6%|▌         | 10/166 [00:56<15:43,  6.05s/it]

  7%|▋         | 11/166 [01:02<15:16,  5.91s/it]

  7%|▋         | 12/166 [01:08<15:14,  5.94s/it]

  8%|▊         | 13/166 [01:14<15:33,  6.10s/it]

  8%|▊         | 14/166 [01:22<16:22,  6.46s/it]

  9%|▉         | 15/166 [01:27<15:40,  6.23s/it]

 10%|▉         | 16/166 [01:35<16:23,  6.56s/it]

 10%|█         | 17/166 [01:41<16:23,  6.60s/it]

 11%|█         | 18/166 [01:48<16:05,  6.53s/it]

 11%|█▏        | 19/166 [01:54<15:48,  6.45s/it]

 12%|█▏        | 20/166 [02:05<18:57,  7.79s/it]

 13%|█▎        | 21/166 [02:12<18:24,  7.61s/it]

 13%|█▎        | 22/166 [02:19<18:03,  7.52s/it]

 14%|█▍        | 23/166 [02:27<18:00,  7.56s/it]

 14%|█▍        | 24/166 [02:33<17:07,  7.23s/it]

 15%|█▌        | 25/166 [02:41<16:58,  7.22s/it]

 16%|█▌        | 26/166 [02:48<16:50,  7.22s/it]

 16%|█▋        | 27/166 [02:55<16:49,  7.27s/it]

 17%|█▋        | 28/166 [03:01<15:58,  6.95s/it]

 17%|█▋        | 29/166 [03:09<16:35,  7.27s/it]

 18%|█▊        | 30/166 [03:16<16:16,  7.18s/it]

 19%|█▊        | 31/166 [03:24<16:12,  7.21s/it]

 19%|█▉        | 32/166 [03:32<16:38,  7.45s/it]

 20%|█▉        | 33/166 [03:39<16:23,  7.40s/it]

 20%|██        | 34/166 [03:45<15:37,  7.10s/it]

 21%|██        | 35/166 [03:54<16:17,  7.46s/it]

 22%|██▏       | 36/166 [04:01<16:09,  7.46s/it]

 22%|██▏       | 37/166 [04:09<16:02,  7.46s/it]

 23%|██▎       | 38/166 [04:18<17:18,  8.12s/it]

 23%|██▎       | 39/166 [04:25<16:24,  7.75s/it]

 24%|██▍       | 40/166 [04:33<16:14,  7.74s/it]

 25%|██▍       | 41/166 [04:40<15:35,  7.48s/it]

 25%|██▌       | 42/166 [04:46<14:58,  7.25s/it]

 26%|██▌       | 43/166 [04:53<14:20,  7.00s/it]

 27%|██▋       | 44/166 [05:02<15:16,  7.51s/it]

 27%|██▋       | 45/166 [05:10<15:29,  7.68s/it]

 28%|██▊       | 46/166 [05:20<17:05,  8.54s/it]

 28%|██▊       | 47/166 [05:28<16:44,  8.44s/it]

 29%|██▉       | 48/166 [05:36<15:48,  8.04s/it]

 30%|██▉       | 49/166 [05:41<14:24,  7.39s/it]

 30%|███       | 50/166 [05:50<15:11,  7.86s/it]

 31%|███       | 51/166 [05:58<14:49,  7.73s/it]

 31%|███▏      | 52/166 [06:05<14:38,  7.71s/it]

 32%|███▏      | 53/166 [06:13<14:12,  7.54s/it]

 33%|███▎      | 54/166 [06:21<14:30,  7.78s/it]

 33%|███▎      | 55/166 [06:27<13:21,  7.22s/it]

 34%|███▎      | 56/166 [06:33<12:40,  6.91s/it]

 34%|███▍      | 57/166 [06:41<13:15,  7.30s/it]

 35%|███▍      | 58/166 [06:47<12:11,  6.77s/it]

 36%|███▌      | 59/166 [06:54<12:23,  6.95s/it]

 36%|███▌      | 60/166 [07:01<12:21,  6.99s/it]

 37%|███▋      | 61/166 [07:10<13:21,  7.63s/it]

 37%|███▋      | 62/166 [07:18<13:04,  7.54s/it]

 38%|███▊      | 63/166 [07:25<12:56,  7.53s/it]

 39%|███▊      | 64/166 [07:34<13:39,  8.03s/it]

 39%|███▉      | 65/166 [07:43<13:40,  8.12s/it]

 40%|███▉      | 66/166 [07:50<12:54,  7.74s/it]

 40%|████      | 67/166 [07:57<12:32,  7.60s/it]

 41%|████      | 68/166 [08:02<11:18,  6.92s/it]

 42%|████▏     | 69/166 [08:09<11:09,  6.90s/it]

 42%|████▏     | 70/166 [08:17<11:34,  7.24s/it]

 43%|████▎     | 71/166 [08:25<11:45,  7.43s/it]

 43%|████▎     | 72/166 [08:31<11:11,  7.14s/it]

 44%|████▍     | 73/166 [08:38<10:43,  6.92s/it]

 45%|████▍     | 74/166 [08:44<10:19,  6.74s/it]

 45%|████▌     | 75/166 [08:50<10:01,  6.61s/it]

 46%|████▌     | 76/166 [08:58<10:10,  6.79s/it]

 46%|████▋     | 77/166 [09:03<09:37,  6.49s/it]

 47%|████▋     | 78/166 [09:11<09:47,  6.67s/it]

 48%|████▊     | 79/166 [09:19<10:13,  7.05s/it]

 48%|████▊     | 80/166 [09:27<10:38,  7.42s/it]

 49%|████▉     | 81/166 [09:35<10:40,  7.54s/it]

 49%|████▉     | 82/166 [09:42<10:17,  7.35s/it]

 50%|█████     | 83/166 [09:47<09:26,  6.83s/it]

 51%|█████     | 84/166 [09:54<09:11,  6.73s/it]

 51%|█████     | 85/166 [10:01<09:18,  6.89s/it]

 52%|█████▏    | 86/166 [10:09<09:45,  7.32s/it]

 52%|█████▏    | 87/166 [10:17<09:44,  7.40s/it]

 53%|█████▎    | 88/166 [10:24<09:31,  7.32s/it]

 54%|█████▎    | 89/166 [10:31<09:23,  7.32s/it]

 54%|█████▍    | 90/166 [10:38<08:58,  7.09s/it]

 55%|█████▍    | 91/166 [10:44<08:41,  6.95s/it]

 55%|█████▌    | 92/166 [10:50<08:11,  6.64s/it]

 56%|█████▌    | 93/166 [10:59<08:47,  7.23s/it]

 57%|█████▋    | 94/166 [11:09<09:43,  8.11s/it]

 57%|█████▋    | 95/166 [11:14<08:34,  7.25s/it]

 58%|█████▊    | 96/166 [11:21<08:11,  7.02s/it]

 58%|█████▊    | 97/166 [11:31<09:15,  8.05s/it]

 59%|█████▉    | 98/166 [11:41<09:44,  8.60s/it]

 60%|█████▉    | 99/166 [11:49<09:13,  8.26s/it]

 60%|██████    | 100/166 [11:55<08:24,  7.64s/it]

 61%|██████    | 101/166 [12:02<08:03,  7.43s/it]

 61%|██████▏   | 102/166 [12:11<08:23,  7.87s/it]

 62%|██████▏   | 103/166 [12:20<08:36,  8.20s/it]

 63%|██████▎   | 104/166 [12:27<08:10,  7.91s/it]

 63%|██████▎   | 105/166 [12:36<08:15,  8.12s/it]

 64%|██████▍   | 106/166 [12:45<08:37,  8.63s/it]

 64%|██████▍   | 107/166 [12:55<08:49,  8.97s/it]

 65%|██████▌   | 108/166 [13:01<07:38,  7.91s/it]

 66%|██████▌   | 109/166 [13:07<07:08,  7.51s/it]

 66%|██████▋   | 110/166 [13:15<06:59,  7.49s/it]

 67%|██████▋   | 111/166 [13:21<06:27,  7.04s/it]

 67%|██████▋   | 112/166 [13:28<06:23,  7.11s/it]

 68%|██████▊   | 113/166 [13:37<06:44,  7.63s/it]

 69%|██████▊   | 114/166 [13:44<06:33,  7.57s/it]

 69%|██████▉   | 115/166 [13:50<06:06,  7.19s/it]

 70%|██████▉   | 116/166 [13:57<05:46,  6.92s/it]

 70%|███████   | 117/166 [14:04<05:47,  7.10s/it]

 71%|███████   | 118/166 [14:12<05:54,  7.38s/it]

 72%|███████▏  | 119/166 [14:20<05:51,  7.48s/it]

 72%|███████▏  | 120/166 [14:28<05:54,  7.70s/it]

 73%|███████▎  | 121/166 [14:36<05:54,  7.89s/it]

 73%|███████▎  | 122/166 [14:45<05:48,  7.93s/it]

 74%|███████▍  | 123/166 [14:52<05:32,  7.73s/it]

 75%|███████▍  | 124/166 [14:58<05:11,  7.42s/it]

 75%|███████▌  | 125/166 [15:06<05:04,  7.42s/it]

 76%|███████▌  | 126/166 [15:15<05:13,  7.84s/it]

 77%|███████▋  | 127/166 [15:20<04:35,  7.05s/it]

 77%|███████▋  | 128/166 [15:30<05:02,  7.96s/it]

 78%|███████▊  | 129/166 [15:38<04:53,  7.94s/it]

 78%|███████▊  | 130/166 [15:45<04:34,  7.62s/it]

 79%|███████▉  | 131/166 [15:52<04:23,  7.54s/it]

 80%|███████▉  | 132/166 [16:00<04:22,  7.72s/it]

 80%|████████  | 133/166 [16:08<04:14,  7.71s/it]

 81%|████████  | 134/166 [16:15<03:55,  7.36s/it]

 81%|████████▏ | 135/166 [16:21<03:36,  6.99s/it]

 82%|████████▏ | 136/166 [16:32<04:05,  8.18s/it]

 83%|████████▎ | 137/166 [16:41<04:04,  8.43s/it]

 83%|████████▎ | 138/166 [16:51<04:09,  8.93s/it]

 84%|████████▎ | 139/166 [16:59<03:58,  8.84s/it]

 84%|████████▍ | 140/166 [17:06<03:30,  8.08s/it]

 85%|████████▍ | 141/166 [17:12<03:05,  7.44s/it]

 86%|████████▌ | 142/166 [17:19<02:56,  7.35s/it]

 86%|████████▌ | 143/166 [17:24<02:36,  6.80s/it]

 87%|████████▋ | 144/166 [17:33<02:45,  7.52s/it]

 87%|████████▋ | 145/166 [17:43<02:51,  8.18s/it]

 88%|████████▊ | 146/166 [17:53<02:50,  8.53s/it]

 89%|████████▊ | 147/166 [18:01<02:41,  8.49s/it]

 89%|████████▉ | 148/166 [18:10<02:35,  8.62s/it]

 90%|████████▉ | 149/166 [18:19<02:29,  8.79s/it]

 90%|█████████ | 150/166 [18:29<02:25,  9.09s/it]

 91%|█████████ | 151/166 [18:37<02:12,  8.83s/it]

 92%|█████████▏| 152/166 [18:43<01:50,  7.91s/it]

 92%|█████████▏| 153/166 [18:49<01:37,  7.47s/it]

 93%|█████████▎| 154/166 [18:57<01:29,  7.43s/it]

 93%|█████████▎| 155/166 [19:04<01:22,  7.53s/it]

 94%|█████████▍| 156/166 [19:10<01:11,  7.11s/it]

 95%|█████████▍| 157/166 [19:17<01:02,  6.99s/it]

 95%|█████████▌| 158/166 [19:23<00:52,  6.57s/it]

 96%|█████████▌| 159/166 [19:28<00:42,  6.08s/it]

 96%|█████████▋| 160/166 [19:33<00:34,  5.77s/it]

 97%|█████████▋| 161/166 [19:39<00:29,  5.86s/it]

 98%|█████████▊| 162/166 [19:46<00:25,  6.28s/it]

 98%|█████████▊| 163/166 [19:53<00:19,  6.55s/it]

 99%|█████████▉| 164/166 [19:59<00:12,  6.44s/it]

 99%|█████████▉| 165/166 [20:05<00:06,  6.29s/it]

100%|██████████| 166/166 [20:10<00:00,  5.64s/it]
                                                       

                                                 
{'eval_loss': 0.25634765625, 'eval_wer': 8.37737794884072, 'eval_runtime': 1289.0888, 'eval_samples_per_second': 1.028, 'eval_steps_per_second': 0.129, 'epoch': 48.77}

 40%|████      | 2000/5000 [6:57:21<9:14:57, 11.10s/it]

100%|██████████| 166/166 [21:23<00:00,  5.64s/it]

                                                 [INFO|trainer.py:2700] 2022-12-20 17:19:45,604 >> Saving model checkpoint to ./checkpoint-2000
[INFO|configuration_utils.py:447] 2022-12-20 17:19:45,605 >> Configuration saved in ./checkpoint-2000/config.json
[INFO|modeling_utils.py:1680] 2022-12-20 17:19:49,244 >> Model weights saved in ./checkpoint-2000/pytorch_model.bin
[INFO|feature_extraction_utils.py:368] 2022-12-20 17:19:49,259 >> Feature extractor saved in ./checkpoint-2000/preprocessor_config.json
[2022-12-20 17:19:49,260] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step2024 is begin to save!
/home/milan/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
[2022-12-20 17:19:49,270] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-2000/global_step2024/mp_rank_00_model_states.pt
[2022-12-20 17:19:49,270] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-2000/global_step2024/mp_rank_00_model_states.pt...
[2022-12-20 17:19:52,982] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-2000/global_step2024/mp_rank_00_model_states.pt.
[2022-12-20 17:19:52,984] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-2000/global_step2024/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2022-12-20 17:20:08,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-2000/global_step2024/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2022-12-20 17:20:08,330] [INFO] [engine.py:3269:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-2000/global_step2024/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2022-12-20 17:20:08,330] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2024 is ready now!
[INFO|feature_extraction_utils.py:368] 2022-12-20 17:20:13,382 >> Feature extractor saved in ./preprocessor_config.json
[2022-12-20 17:22:16,998] [INFO] [timer.py:197:stop] 0/4050, RunningAvgSamplesPerSec=5.869386070314113, CurrSamplesPerSec=5.068967296185505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2001/5000 [6:59:52<366:28:31, 439.92s/it][2022-12-20 17:22:28,005] [INFO] [timer.py:197:stop] 0/4052, RunningAvgSamplesPerSec=5.869391927530963, CurrSamplesPerSec=5.300500373019847, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2002/5000 [7:00:03<259:11:50, 311.24s/it][2022-12-20 17:22:39,002] [INFO] [timer.py:197:stop] 0/4054, RunningAvgSamplesPerSec=5.869400390538758, CurrSamplesPerSec=5.325949626901429, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2003/5000 [7:00:14<184:07:26, 221.17s/it][2022-12-20 17:22:49,982] [INFO] [timer.py:197:stop] 0/4056, RunningAvgSamplesPerSec=5.869413120329157, CurrSamplesPerSec=5.35738468026286, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2004/5000 [7:00:25<131:35:06, 158.11s/it][2022-12-20 17:23:01,364] [INFO] [timer.py:197:stop] 0/4058, RunningAvgSamplesPerSec=5.869319147153795, CurrSamplesPerSec=5.026528964955302, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2005/5000 [7:00:36<94:55:11, 114.09s/it] [2022-12-20 17:23:12,303] [INFO] [logging.py:68:log_dist] [Rank 0] step=2030, skipped=3, lr=[6.608888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:23:12,305] [INFO] [timer.py:197:stop] 0/4060, RunningAvgSamplesPerSec=5.8693435398836895, CurrSamplesPerSec=5.35441881870314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2006/5000 [7:00:47<69:09:04, 83.15s/it] [2022-12-20 17:23:23,344] [INFO] [timer.py:197:stop] 0/4062, RunningAvgSamplesPerSec=5.869340453271744, CurrSamplesPerSec=5.3007804664473515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2007/5000 [7:00:58<51:08:34, 61.52s/it][2022-12-20 17:23:34,500] [INFO] [timer.py:197:stop] 0/4064, RunningAvgSamplesPerSec=5.869305965318879, CurrSamplesPerSec=5.1783726478217575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2008/5000 [7:01:10<38:34:11, 46.41s/it][2022-12-20 17:23:45,484] [INFO] [timer.py:197:stop] 0/4066, RunningAvgSamplesPerSec=5.869317420587939, CurrSamplesPerSec=5.322756063620821, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2009/5000 [7:01:21<29:43:39, 35.78s/it][2022-12-20 17:23:55,607] [INFO] [timer.py:197:stop] 0/4068, RunningAvgSamplesPerSec=5.869554713583558, CurrSamplesPerSec=5.306858742725285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2010/5000 [7:01:36<24:32:51, 29.56s/it][2022-12-20 17:24:06,588] [INFO] [timer.py:197:stop] 0/4070, RunningAvgSamplesPerSec=5.869566827895841, CurrSamplesPerSec=5.325485350699582, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2011/5000 [7:01:47<19:54:55, 23.99s/it][2022-12-20 17:24:17,566] [INFO] [timer.py:197:stop] 0/4072, RunningAvgSamplesPerSec=5.869579598936879, CurrSamplesPerSec=5.33692707881441, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2012/5000 [7:01:58<16:40:17, 20.09s/it][2022-12-20 17:24:28,607] [INFO] [timer.py:197:stop] 0/4074, RunningAvgSamplesPerSec=5.869575694735896, CurrSamplesPerSec=5.287936732481226, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2013/5000 [7:02:09<14:25:10, 17.38s/it][2022-12-20 17:24:39,601] [INFO] [timer.py:197:stop] 0/4076, RunningAvgSamplesPerSec=5.869584134818115, CurrSamplesPerSec=5.347553982542901, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2014/5000 [7:02:20<12:49:27, 15.46s/it][2022-12-20 17:24:50,620] [INFO] [timer.py:197:stop] 0/4078, RunningAvgSamplesPerSec=5.869585991184868, CurrSamplesPerSec=5.319675120965998, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2015/5000 [7:02:31<11:43:01, 14.13s/it][2022-12-20 17:25:01,601] [INFO] [logging.py:68:log_dist] [Rank 0] step=2040, skipped=3, lr=[6.5866666666666666e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:25:01,603] [INFO] [timer.py:197:stop] 0/4080, RunningAvgSamplesPerSec=5.869597521017075, CurrSamplesPerSec=5.358913882848374, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2016/5000 [7:02:42<10:55:10, 13.17s/it][2022-12-20 17:25:12,669] [INFO] [timer.py:197:stop] 0/4082, RunningAvgSamplesPerSec=5.869587089116151, CurrSamplesPerSec=5.248799867131394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2017/5000 [7:02:53<10:23:58, 12.55s/it][2022-12-20 17:25:23,670] [INFO] [timer.py:197:stop] 0/4084, RunningAvgSamplesPerSec=5.8695936001170415, CurrSamplesPerSec=5.332328985556303, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2018/5000 [7:03:04<10:00:53, 12.09s/it][2022-12-20 17:25:34,708] [INFO] [timer.py:197:stop] 0/4086, RunningAvgSamplesPerSec=5.869590693513245, CurrSamplesPerSec=5.313192001828257, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2019/5000 [7:03:15<9:44:35, 11.77s/it] [2022-12-20 17:25:45,732] [INFO] [timer.py:197:stop] 0/4088, RunningAvgSamplesPerSec=5.8695919667493515, CurrSamplesPerSec=5.3033937948372945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2020/5000 [7:03:26<9:33:43, 11.55s/it][2022-12-20 17:25:56,763] [INFO] [timer.py:197:stop] 0/4090, RunningAvgSamplesPerSec=5.869590714367239, CurrSamplesPerSec=5.318152423079144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2021/5000 [7:03:37<9:26:09, 11.40s/it][2022-12-20 17:26:07,777] [INFO] [timer.py:197:stop] 0/4092, RunningAvgSamplesPerSec=5.869594352964098, CurrSamplesPerSec=5.357536941066056, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2022/5000 [7:03:48<9:19:48, 11.28s/it][2022-12-20 17:26:18,833] [INFO] [timer.py:197:stop] 0/4094, RunningAvgSamplesPerSec=5.86958656563254, CurrSamplesPerSec=5.297363382387859, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2023/5000 [7:03:59<9:15:52, 11.20s/it][2022-12-20 17:26:29,842] [INFO] [timer.py:197:stop] 0/4096, RunningAvgSamplesPerSec=5.86959094824509, CurrSamplesPerSec=5.312335466008394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2024/5000 [7:04:10<9:13:08, 11.15s/it][2022-12-20 17:26:40,823] [INFO] [timer.py:197:stop] 0/4098, RunningAvgSamplesPerSec=5.86960310088504, CurrSamplesPerSec=5.3579978393593795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 40%|████      | 2025/5000 [7:04:21<9:11:07, 11.12s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.566666666666667e-06, 'epoch': 49.39}

 40%|████      | 2025/5000 [7:04:21<9:11:07, 11.12s/it][2022-12-20 17:26:51,964] [INFO] [logging.py:68:log_dist] [Rank 0] step=2050, skipped=3, lr=[6.564444444444446e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:26:51,966] [INFO] [timer.py:197:stop] 0/4100, RunningAvgSamplesPerSec=5.869589859741664, CurrSamplesPerSec=5.314459542150491, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2026/5000 [7:04:32<9:10:45, 11.11s/it][2022-12-20 17:27:02,990] [INFO] [timer.py:197:stop] 0/4102, RunningAvgSamplesPerSec=5.869590172591566, CurrSamplesPerSec=5.3264220164408345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2027/5000 [7:04:43<9:09:34, 11.09s/it][2022-12-20 17:27:14,016] [INFO] [timer.py:197:stop] 0/4104, RunningAvgSamplesPerSec=5.869593641112581, CurrSamplesPerSec=5.3428252687032405, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2028/5000 [7:04:54<9:07:58, 11.06s/it][2022-12-20 17:27:25,099] [INFO] [timer.py:197:stop] 0/4106, RunningAvgSamplesPerSec=5.869592780393165, CurrSamplesPerSec=5.312320537437671, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2029/5000 [7:05:05<9:07:46, 11.06s/it][2022-12-20 17:27:36,072] [INFO] [timer.py:197:stop] 0/4108, RunningAvgSamplesPerSec=5.869606700689769, CurrSamplesPerSec=5.345410613300867, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2030/5000 [7:05:16<9:07:16, 11.06s/it][2022-12-20 17:27:47,148] [INFO] [timer.py:197:stop] 0/4110, RunningAvgSamplesPerSec=5.869595055009693, CurrSamplesPerSec=5.314151279604911, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2031/5000 [7:05:27<9:07:03, 11.06s/it][2022-12-20 17:27:58,184] [INFO] [timer.py:197:stop] 0/4112, RunningAvgSamplesPerSec=5.869594091246176, CurrSamplesPerSec=5.3348206322016525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2032/5000 [7:05:38<9:06:16, 11.04s/it][2022-12-20 17:28:09,233] [INFO] [timer.py:197:stop] 0/4114, RunningAvgSamplesPerSec=5.869588120567921, CurrSamplesPerSec=5.300064591434987, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2033/5000 [7:05:49<9:06:36, 11.05s/it][2022-12-20 17:28:20,310] [INFO] [timer.py:197:stop] 0/4116, RunningAvgSamplesPerSec=5.869578971512619, CurrSamplesPerSec=5.300665117960555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2034/5000 [7:06:00<9:06:23, 11.05s/it][2022-12-20 17:28:31,321] [INFO] [timer.py:197:stop] 0/4118, RunningAvgSamplesPerSec=5.869585258911873, CurrSamplesPerSec=5.343393829919865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2035/5000 [7:06:11<9:05:29, 11.04s/it][2022-12-20 17:28:42,262] [INFO] [logging.py:68:log_dist] [Rank 0] step=2060, skipped=3, lr=[6.5422222222222235e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:28:42,264] [INFO] [timer.py:197:stop] 0/4120, RunningAvgSamplesPerSec=5.869606882394255, CurrSamplesPerSec=5.391906828586765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2036/5000 [7:06:22<9:04:14, 11.02s/it][2022-12-20 17:28:53,318] [INFO] [timer.py:197:stop] 0/4122, RunningAvgSamplesPerSec=5.869599794371608, CurrSamplesPerSec=5.315206043359304, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2037/5000 [7:06:33<9:03:55, 11.01s/it][2022-12-20 17:29:04,503] [INFO] [timer.py:197:stop] 0/4124, RunningAvgSamplesPerSec=5.869558529313979, CurrSamplesPerSec=5.163125641915442, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2038/5000 [7:06:45<9:06:58, 11.08s/it][2022-12-20 17:29:15,558] [INFO] [timer.py:197:stop] 0/4126, RunningAvgSamplesPerSec=5.8695514107354, CurrSamplesPerSec=5.316132990019348, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2039/5000 [7:06:56<9:05:54, 11.06s/it][2022-12-20 17:29:26,563] [INFO] [timer.py:197:stop] 0/4128, RunningAvgSamplesPerSec=5.869556915876289, CurrSamplesPerSec=5.327956222211054, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2040/5000 [7:07:07<9:05:13, 11.05s/it][2022-12-20 17:29:37,613] [INFO] [timer.py:197:stop] 0/4130, RunningAvgSamplesPerSec=5.8695506649490925, CurrSamplesPerSec=5.309514405792698, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2041/5000 [7:07:18<9:05:00, 11.05s/it][2022-12-20 17:29:48,625] [INFO] [timer.py:197:stop] 0/4132, RunningAvgSamplesPerSec=5.86955452873535, CurrSamplesPerSec=5.342272776320921, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2042/5000 [7:07:29<9:04:17, 11.04s/it][2022-12-20 17:29:59,638] [INFO] [timer.py:197:stop] 0/4134, RunningAvgSamplesPerSec=5.869558156887463, CurrSamplesPerSec=5.345172614523018, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2043/5000 [7:07:40<9:03:32, 11.03s/it][2022-12-20 17:30:10,689] [INFO] [timer.py:197:stop] 0/4136, RunningAvgSamplesPerSec=5.869552224078155, CurrSamplesPerSec=5.302943918581562, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2044/5000 [7:07:51<9:03:48, 11.04s/it][2022-12-20 17:30:21,690] [INFO] [timer.py:197:stop] 0/4138, RunningAvgSamplesPerSec=5.8695625811830645, CurrSamplesPerSec=5.353986727297278, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2045/5000 [7:08:02<9:02:57, 11.02s/it][2022-12-20 17:30:32,739] [INFO] [logging.py:68:log_dist] [Rank 0] step=2070, skipped=3, lr=[6.520000000000001e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:30:32,740] [INFO] [timer.py:197:stop] 0/4140, RunningAvgSamplesPerSec=5.869560732376997, CurrSamplesPerSec=5.316873642806605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2046/5000 [7:08:13<9:03:22, 11.04s/it][2022-12-20 17:30:43,829] [INFO] [timer.py:197:stop] 0/4142, RunningAvgSamplesPerSec=5.869544485222814, CurrSamplesPerSec=5.280961092956325, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2047/5000 [7:08:24<9:03:48, 11.05s/it][2022-12-20 17:30:54,862] [INFO] [timer.py:197:stop] 0/4144, RunningAvgSamplesPerSec=5.86954288194538, CurrSamplesPerSec=5.321929993984476, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2048/5000 [7:08:35<9:03:14, 11.04s/it][2022-12-20 17:31:05,924] [INFO] [timer.py:197:stop] 0/4146, RunningAvgSamplesPerSec=5.869533864641028, CurrSamplesPerSec=5.287930065763785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2049/5000 [7:08:46<9:03:51, 11.06s/it][2022-12-20 17:31:17,158] [INFO] [timer.py:197:stop] 0/4148, RunningAvgSamplesPerSec=5.8694834639432605, CurrSamplesPerSec=5.181297246311236, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2050/5000 [7:08:57<9:08:30, 11.16s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.511111111111112e-06, 'epoch': 49.99}

 41%|████      | 2050/5000 [7:08:57<9:08:30, 11.16s/it][2022-12-20 17:31:27,473] [INFO] [timer.py:197:stop] 0/4150, RunningAvgSamplesPerSec=5.8697079344926655, CurrSamplesPerSec=6.239696643891063, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 17:31:38,466] [INFO] [timer.py:197:stop] 0/4152, RunningAvgSamplesPerSec=5.8697160700537525, CurrSamplesPerSec=5.350369764565395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2051/5000 [7:09:14<10:22:20, 12.66s/it][2022-12-20 17:31:49,495] [INFO] [timer.py:197:stop] 0/4154, RunningAvgSamplesPerSec=5.8697156034390545, CurrSamplesPerSec=5.302351045047399, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2052/5000 [7:09:25<9:58:03, 12.17s/it] [2022-12-20 17:32:00,602] [INFO] [timer.py:197:stop] 0/4156, RunningAvgSamplesPerSec=5.869694646721922, CurrSamplesPerSec=5.338214675430894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2053/5000 [7:09:36<9:42:09, 11.85s/it][2022-12-20 17:32:11,641] [INFO] [timer.py:197:stop] 0/4158, RunningAvgSamplesPerSec=5.86969148264135, CurrSamplesPerSec=5.290477350090553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2054/5000 [7:09:47<9:29:58, 11.61s/it][2022-12-20 17:32:22,663] [INFO] [logging.py:68:log_dist] [Rank 0] step=2080, skipped=3, lr=[6.497777777777779e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:32:22,665] [INFO] [timer.py:197:stop] 0/4160, RunningAvgSamplesPerSec=5.869692209595589, CurrSamplesPerSec=5.329147236245231, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2055/5000 [7:09:58<9:21:10, 11.43s/it][2022-12-20 17:32:33,753] [INFO] [timer.py:197:stop] 0/4162, RunningAvgSamplesPerSec=5.86967641522077, CurrSamplesPerSec=5.338710054012893, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2056/5000 [7:10:09<9:15:54, 11.33s/it][2022-12-20 17:32:44,769] [INFO] [timer.py:197:stop] 0/4164, RunningAvgSamplesPerSec=5.869679360293293, CurrSamplesPerSec=5.312447958640891, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2057/5000 [7:10:20<9:11:06, 11.24s/it][2022-12-20 17:32:55,779] [INFO] [timer.py:197:stop] 0/4166, RunningAvgSamplesPerSec=5.869683787419864, CurrSamplesPerSec=5.353371710050705, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2058/5000 [7:10:31<9:07:35, 11.17s/it][2022-12-20 17:33:06,788] [INFO] [timer.py:197:stop] 0/4168, RunningAvgSamplesPerSec=5.869688566079263, CurrSamplesPerSec=5.343523171623795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2059/5000 [7:10:42<9:05:04, 11.12s/it][2022-12-20 17:33:17,843] [INFO] [timer.py:197:stop] 0/4170, RunningAvgSamplesPerSec=5.869681574305433, CurrSamplesPerSec=5.309039970327098, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2060/5000 [7:10:53<9:03:55, 11.10s/it][2022-12-20 17:33:28,864] [INFO] [timer.py:197:stop] 0/4172, RunningAvgSamplesPerSec=5.869683134701962, CurrSamplesPerSec=5.324704695455814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2061/5000 [7:11:04<9:02:34, 11.08s/it][2022-12-20 17:33:39,971] [INFO] [timer.py:197:stop] 0/4174, RunningAvgSamplesPerSec=5.869662549136632, CurrSamplesPerSec=5.330508775134229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████      | 2062/5000 [7:11:15<9:02:50, 11.09s/it][2022-12-20 17:33:50,954] [INFO] [timer.py:197:stop] 0/4176, RunningAvgSamplesPerSec=5.86967388188391, CurrSamplesPerSec=5.33916028488574, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2063/5000 [7:11:26<9:01:08, 11.06s/it][2022-12-20 17:34:02,016] [INFO] [timer.py:197:stop] 0/4178, RunningAvgSamplesPerSec=5.869665145304002, CurrSamplesPerSec=5.353101189922566, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2064/5000 [7:11:37<9:01:03, 11.06s/it][2022-12-20 17:34:13,118] [INFO] [logging.py:68:log_dist] [Rank 0] step=2090, skipped=3, lr=[6.475555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:34:13,120] [INFO] [timer.py:197:stop] 0/4180, RunningAvgSamplesPerSec=5.869645570805209, CurrSamplesPerSec=5.302852569708583, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2065/5000 [7:11:48<9:01:33, 11.07s/it][2022-12-20 17:34:24,192] [INFO] [timer.py:197:stop] 0/4182, RunningAvgSamplesPerSec=5.869634215174035, CurrSamplesPerSec=5.314287836552514, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2066/5000 [7:11:59<9:01:23, 11.07s/it][2022-12-20 17:34:35,265] [INFO] [timer.py:197:stop] 0/4184, RunningAvgSamplesPerSec=5.869622344729002, CurrSamplesPerSec=5.2766248403265195, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2067/5000 [7:12:10<9:01:13, 11.07s/it][2022-12-20 17:34:46,238] [INFO] [timer.py:197:stop] 0/4186, RunningAvgSamplesPerSec=5.869636679265411, CurrSamplesPerSec=5.3299302530564425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2068/5000 [7:12:21<8:59:35, 11.04s/it][2022-12-20 17:34:57,222] [INFO] [timer.py:197:stop] 0/4188, RunningAvgSamplesPerSec=5.86964808370703, CurrSamplesPerSec=5.367462580490307, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2069/5000 [7:12:32<8:58:33, 11.02s/it][2022-12-20 17:35:08,234] [INFO] [timer.py:197:stop] 0/4190, RunningAvgSamplesPerSec=5.8696522614952, CurrSamplesPerSec=5.348995290875572, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2070/5000 [7:12:43<8:58:11, 11.02s/it][2022-12-20 17:35:19,554] [INFO] [timer.py:197:stop] 0/4192, RunningAvgSamplesPerSec=5.869577219664674, CurrSamplesPerSec=5.345937990211932, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2071/5000 [7:12:55<9:02:23, 11.11s/it][2022-12-20 17:35:30,545] [INFO] [timer.py:197:stop] 0/4194, RunningAvgSamplesPerSec=5.869586753379504, CurrSamplesPerSec=5.326354798856263, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2072/5000 [7:13:06<9:00:26, 11.07s/it][2022-12-20 17:35:41,542] [INFO] [timer.py:197:stop] 0/4196, RunningAvgSamplesPerSec=5.869594752712661, CurrSamplesPerSec=5.313466917308831, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2073/5000 [7:13:17<8:59:07, 11.05s/it][2022-12-20 17:35:52,612] [INFO] [timer.py:197:stop] 0/4198, RunningAvgSamplesPerSec=5.869583978725496, CurrSamplesPerSec=5.298326573661605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 41%|████▏     | 2074/5000 [7:13:28<8:59:12, 11.06s/it][2022-12-20 17:36:03,607] [INFO] [logging.py:68:log_dist] [Rank 0] step=2100, skipped=3, lr=[6.453333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:36:03,608] [INFO] [timer.py:197:stop] 0/4200, RunningAvgSamplesPerSec=5.869591912806813, CurrSamplesPerSec=5.348805998925758, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2075/5000 [7:13:39<8:58:08, 11.04s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.453333333333334e-06, 'epoch': 50.6}

 42%|████▏     | 2075/5000 [7:13:39<8:58:08, 11.04s/it][2022-12-20 17:36:14,626] [INFO] [timer.py:197:stop] 0/4202, RunningAvgSamplesPerSec=5.869594295978162, CurrSamplesPerSec=5.324807150009601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2076/5000 [7:13:50<8:57:39, 11.03s/it][2022-12-20 17:36:25,803] [INFO] [timer.py:197:stop] 0/4204, RunningAvgSamplesPerSec=5.869556074177701, CurrSamplesPerSec=5.3490257749533265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2077/5000 [7:14:01<8:59:34, 11.08s/it][2022-12-20 17:36:36,843] [INFO] [timer.py:197:stop] 0/4206, RunningAvgSamplesPerSec=5.86955279584577, CurrSamplesPerSec=5.311405429106456, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2078/5000 [7:14:12<8:58:51, 11.06s/it][2022-12-20 17:36:47,774] [INFO] [timer.py:197:stop] 0/4208, RunningAvgSamplesPerSec=5.869577425901459, CurrSamplesPerSec=5.386259492479917, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2079/5000 [7:14:23<8:56:43, 11.02s/it][2022-12-20 17:36:58,749] [INFO] [timer.py:197:stop] 0/4210, RunningAvgSamplesPerSec=5.869590692885201, CurrSamplesPerSec=5.365074389402667, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2080/5000 [7:14:34<8:55:49, 11.01s/it][2022-12-20 17:37:09,799] [INFO] [timer.py:197:stop] 0/4212, RunningAvgSamplesPerSec=5.8695851040138844, CurrSamplesPerSec=5.3009906606274955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2081/5000 [7:14:45<8:56:13, 11.02s/it][2022-12-20 17:37:20,877] [INFO] [timer.py:197:stop] 0/4214, RunningAvgSamplesPerSec=5.869572173135971, CurrSamplesPerSec=5.287375953645529, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2082/5000 [7:14:56<8:56:51, 11.04s/it][2022-12-20 17:37:31,891] [INFO] [timer.py:197:stop] 0/4216, RunningAvgSamplesPerSec=5.869575762964075, CurrSamplesPerSec=5.340546920733833, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2083/5000 [7:15:07<8:56:18, 11.03s/it][2022-12-20 17:37:42,909] [INFO] [timer.py:197:stop] 0/4218, RunningAvgSamplesPerSec=5.8695779670188255, CurrSamplesPerSec=5.308307796573596, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2084/5000 [7:15:18<8:55:55, 11.03s/it][2022-12-20 17:37:53,944] [INFO] [logging.py:68:log_dist] [Rank 0] step=2110, skipped=3, lr=[6.4311111111111116e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:37:53,945] [INFO] [timer.py:197:stop] 0/4220, RunningAvgSamplesPerSec=5.8695766717611395, CurrSamplesPerSec=5.322517545035062, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2085/5000 [7:15:29<8:55:52, 11.03s/it][2022-12-20 17:38:04,983] [INFO] [timer.py:197:stop] 0/4222, RunningAvgSamplesPerSec=5.869574562234351, CurrSamplesPerSec=5.306864827768524, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2086/5000 [7:15:40<8:55:48, 11.03s/it][2022-12-20 17:38:15,999] [INFO] [timer.py:197:stop] 0/4224, RunningAvgSamplesPerSec=5.869577307558116, CurrSamplesPerSec=5.307349367899886, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2087/5000 [7:15:51<8:55:22, 11.03s/it][2022-12-20 17:38:27,007] [INFO] [timer.py:197:stop] 0/4226, RunningAvgSamplesPerSec=5.869582315865934, CurrSamplesPerSec=5.345965245459716, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2088/5000 [7:16:02<8:54:54, 11.02s/it][2022-12-20 17:38:38,058] [INFO] [timer.py:197:stop] 0/4228, RunningAvgSamplesPerSec=5.869576059396264, CurrSamplesPerSec=5.3052649420324185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2089/5000 [7:16:13<8:55:09, 11.03s/it][2022-12-20 17:38:49,117] [INFO] [timer.py:197:stop] 0/4230, RunningAvgSamplesPerSec=5.869567975143271, CurrSamplesPerSec=5.294105433290934, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2090/5000 [7:16:24<8:55:23, 11.04s/it][2022-12-20 17:39:00,145] [INFO] [timer.py:197:stop] 0/4232, RunningAvgSamplesPerSec=5.869567600646852, CurrSamplesPerSec=5.319191911425831, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2091/5000 [7:16:35<8:55:02, 11.04s/it][2022-12-20 17:39:10,260] [INFO] [timer.py:197:stop] 0/4234, RunningAvgSamplesPerSec=5.86979676285488, CurrSamplesPerSec=5.327729714798093, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2092/5000 [7:16:50<9:53:39, 12.25s/it][2022-12-20 17:39:21,339] [INFO] [timer.py:197:stop] 0/4236, RunningAvgSamplesPerSec=5.869783350627567, CurrSamplesPerSec=5.288677051016541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2093/5000 [7:17:01<9:36:08, 11.89s/it][2022-12-20 17:39:32,329] [INFO] [timer.py:197:stop] 0/4238, RunningAvgSamplesPerSec=5.869792256322182, CurrSamplesPerSec=5.348110551324939, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2094/5000 [7:17:12<9:23:02, 11.63s/it][2022-12-20 17:39:43,297] [INFO] [logging.py:68:log_dist] [Rank 0] step=2120, skipped=3, lr=[6.408888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:39:43,298] [INFO] [timer.py:197:stop] 0/4240, RunningAvgSamplesPerSec=5.86980670245358, CurrSamplesPerSec=5.378837255398425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2095/5000 [7:17:23<9:15:09, 11.47s/it][2022-12-20 17:39:54,431] [INFO] [timer.py:197:stop] 0/4242, RunningAvgSamplesPerSec=5.869779694916928, CurrSamplesPerSec=5.346042328211769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2096/5000 [7:17:34<9:08:09, 11.33s/it][2022-12-20 17:40:05,494] [INFO] [timer.py:197:stop] 0/4244, RunningAvgSamplesPerSec=5.869770364546634, CurrSamplesPerSec=5.288226125886814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2097/5000 [7:17:46<9:04:16, 11.25s/it][2022-12-20 17:40:16,546] [INFO] [timer.py:197:stop] 0/4246, RunningAvgSamplesPerSec=5.869763862379358, CurrSamplesPerSec=5.305539457228038, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2098/5000 [7:17:57<9:04:25, 11.26s/it][2022-12-20 17:40:27,819] [INFO] [timer.py:197:stop] 0/4248, RunningAvgSamplesPerSec=5.8697012104858315, CurrSamplesPerSec=5.305066360894888, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2099/5000 [7:18:08<9:00:51, 11.19s/it][2022-12-20 17:40:38,806] [INFO] [timer.py:197:stop] 0/4250, RunningAvgSamplesPerSec=5.869711315650282, CurrSamplesPerSec=5.336993714631181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2100/5000 [7:18:19<8:58:20, 11.14s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.397777777777778e-06, 'epoch': 51.22}

 42%|████▏     | 2100/5000 [7:18:19<8:58:20, 11.14s/it][2022-12-20 17:40:49,803] [INFO] [timer.py:197:stop] 0/4252, RunningAvgSamplesPerSec=5.86971910690254, CurrSamplesPerSec=5.3637182874079326, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2101/5000 [7:18:30<8:59:27, 11.16s/it][2022-12-20 17:41:01,112] [INFO] [timer.py:197:stop] 0/4254, RunningAvgSamplesPerSec=5.869647633610892, CurrSamplesPerSec=5.290069277747488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2102/5000 [7:18:41<8:57:33, 11.13s/it][2022-12-20 17:41:12,152] [INFO] [timer.py:197:stop] 0/4256, RunningAvgSamplesPerSec=5.869644173440285, CurrSamplesPerSec=5.2962623941129126, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2103/5000 [7:18:52<8:56:03, 11.10s/it][2022-12-20 17:41:23,193] [INFO] [timer.py:197:stop] 0/4258, RunningAvgSamplesPerSec=5.869640370871527, CurrSamplesPerSec=5.2940626252218665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2104/5000 [7:19:03<8:55:24, 11.09s/it][2022-12-20 17:41:34,251] [INFO] [logging.py:68:log_dist] [Rank 0] step=2130, skipped=3, lr=[6.386666666666668e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:41:34,253] [INFO] [timer.py:197:stop] 0/4260, RunningAvgSamplesPerSec=5.869632039320196, CurrSamplesPerSec=5.302904738748303, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2105/5000 [7:19:14<8:54:15, 11.07s/it][2022-12-20 17:41:45,269] [INFO] [timer.py:197:stop] 0/4262, RunningAvgSamplesPerSec=5.86963465097371, CurrSamplesPerSec=5.312507466016119, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2106/5000 [7:19:25<8:53:17, 11.06s/it][2022-12-20 17:41:56,298] [INFO] [timer.py:197:stop] 0/4264, RunningAvgSamplesPerSec=5.869634075739578, CurrSamplesPerSec=5.302530569080753, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2107/5000 [7:19:36<8:53:14, 11.06s/it][2022-12-20 17:42:07,305] [INFO] [timer.py:197:stop] 0/4266, RunningAvgSamplesPerSec=5.869643806783693, CurrSamplesPerSec=5.370656734958366, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2108/5000 [7:19:47<8:51:26, 11.03s/it][2022-12-20 17:42:18,265] [INFO] [timer.py:197:stop] 0/4268, RunningAvgSamplesPerSec=5.869660726214251, CurrSamplesPerSec=5.3446251700586815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2109/5000 [7:19:58<8:50:24, 11.01s/it][2022-12-20 17:42:29,254] [INFO] [timer.py:197:stop] 0/4270, RunningAvgSamplesPerSec=5.869670177358915, CurrSamplesPerSec=5.325067000887526, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2110/5000 [7:20:09<8:50:48, 11.02s/it][2022-12-20 17:42:40,329] [INFO] [timer.py:197:stop] 0/4272, RunningAvgSamplesPerSec=5.869658165513329, CurrSamplesPerSec=5.3021093247715445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2111/5000 [7:20:20<8:50:28, 11.02s/it][2022-12-20 17:42:51,352] [INFO] [timer.py:197:stop] 0/4274, RunningAvgSamplesPerSec=5.869659146614219, CurrSamplesPerSec=5.289885384270645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2112/5000 [7:20:31<8:51:01, 11.03s/it][2022-12-20 17:43:02,370] [INFO] [timer.py:197:stop] 0/4276, RunningAvgSamplesPerSec=5.869661398917182, CurrSamplesPerSec=5.334349295941013, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2113/5000 [7:20:42<8:51:01, 11.04s/it][2022-12-20 17:43:13,473] [INFO] [timer.py:197:stop] 0/4278, RunningAvgSamplesPerSec=5.869646096751135, CurrSamplesPerSec=5.296525945800256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2114/5000 [7:20:54<8:51:23, 11.05s/it][2022-12-20 17:43:24,511] [INFO] [logging.py:68:log_dist] [Rank 0] step=2140, skipped=3, lr=[6.364444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:43:24,513] [INFO] [timer.py:197:stop] 0/4280, RunningAvgSamplesPerSec=5.8696426152506636, CurrSamplesPerSec=5.31262795676282, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2115/5000 [7:21:05<8:51:13, 11.05s/it][2022-12-20 17:43:35,577] [INFO] [timer.py:197:stop] 0/4282, RunningAvgSamplesPerSec=5.8696329184626155, CurrSamplesPerSec=5.299705680115532, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2116/5000 [7:21:16<8:51:37, 11.06s/it][2022-12-20 17:43:46,652] [INFO] [timer.py:197:stop] 0/4284, RunningAvgSamplesPerSec=5.869620942547407, CurrSamplesPerSec=5.312044689371326, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2117/5000 [7:21:27<8:51:15, 11.06s/it][2022-12-20 17:43:57,685] [INFO] [timer.py:197:stop] 0/4286, RunningAvgSamplesPerSec=5.869619288311004, CurrSamplesPerSec=5.324626959373504, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2118/5000 [7:21:38<8:50:04, 11.04s/it][2022-12-20 17:44:08,685] [INFO] [timer.py:197:stop] 0/4288, RunningAvgSamplesPerSec=5.869626023887719, CurrSamplesPerSec=5.3130315249165285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2119/5000 [7:21:49<8:50:19, 11.04s/it][2022-12-20 17:44:19,804] [INFO] [timer.py:197:stop] 0/4290, RunningAvgSamplesPerSec=5.8696028069119865, CurrSamplesPerSec=5.266956575244587, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2120/5000 [7:22:00<8:50:56, 11.06s/it][2022-12-20 17:44:30,838] [INFO] [timer.py:197:stop] 0/4292, RunningAvgSamplesPerSec=5.869601414518833, CurrSamplesPerSec=5.324822148842129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2121/5000 [7:22:11<8:49:55, 11.04s/it][2022-12-20 17:44:41,825] [INFO] [timer.py:197:stop] 0/4294, RunningAvgSamplesPerSec=5.869611346944094, CurrSamplesPerSec=5.339806243947262, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2122/5000 [7:22:22<8:49:28, 11.04s/it][2022-12-20 17:44:52,817] [INFO] [timer.py:197:stop] 0/4296, RunningAvgSamplesPerSec=5.869619988490275, CurrSamplesPerSec=5.369450533729697, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2123/5000 [7:22:33<8:48:30, 11.02s/it][2022-12-20 17:45:03,818] [INFO] [timer.py:197:stop] 0/4298, RunningAvgSamplesPerSec=5.869626629431279, CurrSamplesPerSec=5.353618340621456, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▏     | 2124/5000 [7:22:44<8:47:31, 11.01s/it][2022-12-20 17:45:14,796] [INFO] [logging.py:68:log_dist] [Rank 0] step=2150, skipped=3, lr=[6.342222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:45:14,797] [INFO] [timer.py:197:stop] 0/4300, RunningAvgSamplesPerSec=5.869638326558733, CurrSamplesPerSec=5.341478688961967, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 42%|████▎     | 2125/5000 [7:22:55<8:46:59, 11.00s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.342222222222223e-06, 'epoch': 51.82}

 42%|████▎     | 2125/5000 [7:22:55<8:46:59, 11.00s/it][2022-12-20 17:45:25,809] [INFO] [timer.py:197:stop] 0/4302, RunningAvgSamplesPerSec=5.869642185507423, CurrSamplesPerSec=5.315709792343204, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2126/5000 [7:23:06<8:47:33, 11.01s/it][2022-12-20 17:45:36,888] [INFO] [timer.py:197:stop] 0/4304, RunningAvgSamplesPerSec=5.869629242509666, CurrSamplesPerSec=5.289866620349006, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2127/5000 [7:23:17<8:48:11, 11.03s/it][2022-12-20 17:45:47,875] [INFO] [timer.py:197:stop] 0/4306, RunningAvgSamplesPerSec=5.869639180685798, CurrSamplesPerSec=5.363519378649219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2128/5000 [7:23:28<8:47:09, 11.01s/it][2022-12-20 17:45:58,840] [INFO] [timer.py:197:stop] 0/4308, RunningAvgSamplesPerSec=5.869654527111631, CurrSamplesPerSec=5.36942540134854, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2129/5000 [7:23:39<8:45:48, 10.99s/it][2022-12-20 17:46:09,833] [INFO] [timer.py:197:stop] 0/4310, RunningAvgSamplesPerSec=5.869662953994062, CurrSamplesPerSec=5.315573372827606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2130/5000 [7:23:50<8:45:43, 10.99s/it][2022-12-20 17:46:20,861] [INFO] [timer.py:197:stop] 0/4312, RunningAvgSamplesPerSec=5.86966269078074, CurrSamplesPerSec=5.286640996889408, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2131/5000 [7:24:01<8:47:12, 11.03s/it][2022-12-20 17:46:31,922] [INFO] [timer.py:197:stop] 0/4314, RunningAvgSamplesPerSec=5.869654094542675, CurrSamplesPerSec=5.326235798299472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2132/5000 [7:24:12<8:46:49, 11.02s/it][2022-12-20 17:46:42,001] [INFO] [timer.py:197:stop] 0/4316, RunningAvgSamplesPerSec=5.869887935151488, CurrSamplesPerSec=6.293578115870945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 17:46:52,967] [INFO] [timer.py:197:stop] 0/4318, RunningAvgSamplesPerSec=5.8699023388172, CurrSamplesPerSec=5.347754265774817, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2133/5000 [7:24:28<9:59:30, 12.55s/it][2022-12-20 17:47:03,915] [INFO] [logging.py:68:log_dist] [Rank 0] step=2160, skipped=3, lr=[6.3200000000000005e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:47:03,916] [INFO] [timer.py:197:stop] 0/4320, RunningAvgSamplesPerSec=5.869921522704897, CurrSamplesPerSec=5.353709097849645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2134/5000 [7:24:39<9:36:24, 12.07s/it][2022-12-20 17:47:14,920] [INFO] [timer.py:197:stop] 0/4322, RunningAvgSamplesPerSec=5.869927275788899, CurrSamplesPerSec=5.30320394487293, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2135/5000 [7:24:50<9:20:58, 11.75s/it][2022-12-20 17:47:25,916] [INFO] [timer.py:197:stop] 0/4324, RunningAvgSamplesPerSec=5.869934934749594, CurrSamplesPerSec=5.34697899735475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2136/5000 [7:25:01<9:10:00, 11.52s/it][2022-12-20 17:47:36,941] [INFO] [timer.py:197:stop] 0/4326, RunningAvgSamplesPerSec=5.869935279665259, CurrSamplesPerSec=5.33774380066359, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2137/5000 [7:25:12<9:02:41, 11.37s/it][2022-12-20 17:47:47,959] [INFO] [timer.py:197:stop] 0/4328, RunningAvgSamplesPerSec=5.869937445561894, CurrSamplesPerSec=5.330176633495267, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2138/5000 [7:25:23<8:57:25, 11.27s/it][2022-12-20 17:47:58,931] [INFO] [timer.py:197:stop] 0/4330, RunningAvgSamplesPerSec=5.8699513153885015, CurrSamplesPerSec=5.351406524313069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2139/5000 [7:25:34<8:53:00, 11.18s/it][2022-12-20 17:48:09,925] [INFO] [timer.py:197:stop] 0/4332, RunningAvgSamplesPerSec=5.869959632934795, CurrSamplesPerSec=5.319912963133473, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2140/5000 [7:25:45<8:50:11, 11.12s/it][2022-12-20 17:48:20,985] [INFO] [timer.py:197:stop] 0/4334, RunningAvgSamplesPerSec=5.869951054808173, CurrSamplesPerSec=5.308060494721827, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2141/5000 [7:25:56<8:49:06, 11.10s/it][2022-12-20 17:48:31,969] [INFO] [timer.py:197:stop] 0/4336, RunningAvgSamplesPerSec=5.869961320350865, CurrSamplesPerSec=5.332545714679501, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2142/5000 [7:26:07<8:47:12, 11.07s/it][2022-12-20 17:48:42,953] [INFO] [timer.py:197:stop] 0/4338, RunningAvgSamplesPerSec=5.869971946443739, CurrSamplesPerSec=5.340183355764844, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2143/5000 [7:26:18<8:45:48, 11.04s/it][2022-12-20 17:48:53,975] [INFO] [logging.py:68:log_dist] [Rank 0] step=2170, skipped=3, lr=[6.297777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:48:53,977] [INFO] [timer.py:197:stop] 0/4340, RunningAvgSamplesPerSec=5.869972284016424, CurrSamplesPerSec=5.3027461396701865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2144/5000 [7:26:29<8:45:21, 11.04s/it][2022-12-20 17:49:04,990] [INFO] [timer.py:197:stop] 0/4342, RunningAvgSamplesPerSec=5.869975692059363, CurrSamplesPerSec=5.302321509585663, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2145/5000 [7:26:40<8:44:50, 11.03s/it][2022-12-20 17:49:16,029] [INFO] [timer.py:197:stop] 0/4344, RunningAvgSamplesPerSec=5.869972887456117, CurrSamplesPerSec=5.305878813210336, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2146/5000 [7:26:51<8:44:46, 11.03s/it][2022-12-20 17:49:27,127] [INFO] [timer.py:197:stop] 0/4346, RunningAvgSamplesPerSec=5.869959062222257, CurrSamplesPerSec=5.284625444551737, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2147/5000 [7:27:02<8:45:32, 11.05s/it][2022-12-20 17:49:38,127] [INFO] [timer.py:197:stop] 0/4348, RunningAvgSamplesPerSec=5.869965526700936, CurrSamplesPerSec=5.346384116915314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2148/5000 [7:27:13<8:44:36, 11.04s/it][2022-12-20 17:49:49,132] [INFO] [timer.py:197:stop] 0/4350, RunningAvgSamplesPerSec=5.869970863714276, CurrSamplesPerSec=5.317595540846101, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2149/5000 [7:27:24<8:43:57, 11.03s/it][2022-12-20 17:50:00,078] [INFO] [timer.py:197:stop] 0/4352, RunningAvgSamplesPerSec=5.869990840164583, CurrSamplesPerSec=5.375280292652161, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2150/5000 [7:27:35<8:42:37, 11.00s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.284444444444445e-06, 'epoch': 52.43}

 43%|████▎     | 2150/5000 [7:27:35<8:42:37, 11.00s/it][2022-12-20 17:50:11,118] [INFO] [timer.py:197:stop] 0/4354, RunningAvgSamplesPerSec=5.869987759194347, CurrSamplesPerSec=5.327790199486686, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2151/5000 [7:27:46<8:42:58, 11.01s/it][2022-12-20 17:50:22,097] [INFO] [timer.py:197:stop] 0/4356, RunningAvgSamplesPerSec=5.8699993219459765, CurrSamplesPerSec=5.341547776878396, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2152/5000 [7:27:57<8:42:17, 11.00s/it][2022-12-20 17:50:33,107] [INFO] [timer.py:197:stop] 0/4358, RunningAvgSamplesPerSec=5.870004415816592, CurrSamplesPerSec=5.3410822651618055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2153/5000 [7:28:08<8:42:12, 11.01s/it][2022-12-20 17:50:44,168] [INFO] [logging.py:68:log_dist] [Rank 0] step=2180, skipped=3, lr=[6.275555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:50:44,170] [INFO] [timer.py:197:stop] 0/4360, RunningAvgSamplesPerSec=5.86999523356314, CurrSamplesPerSec=5.284802729842509, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2154/5000 [7:28:19<8:42:50, 11.02s/it][2022-12-20 17:50:55,194] [INFO] [timer.py:197:stop] 0/4362, RunningAvgSamplesPerSec=5.869995830137972, CurrSamplesPerSec=5.332496774305306, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2155/5000 [7:28:30<8:42:40, 11.02s/it][2022-12-20 17:51:06,184] [INFO] [timer.py:197:stop] 0/4364, RunningAvgSamplesPerSec=5.8700047786994265, CurrSamplesPerSec=5.334004804580248, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2156/5000 [7:28:41<8:42:01, 11.01s/it][2022-12-20 17:51:17,164] [INFO] [timer.py:197:stop] 0/4366, RunningAvgSamplesPerSec=5.8700162725724425, CurrSamplesPerSec=5.324683148814321, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2157/5000 [7:28:52<8:41:22, 11.00s/it][2022-12-20 17:51:28,170] [INFO] [timer.py:197:stop] 0/4368, RunningAvgSamplesPerSec=5.870021390200435, CurrSamplesPerSec=5.337694552556263, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2158/5000 [7:29:03<8:41:13, 11.00s/it][2022-12-20 17:51:39,236] [INFO] [timer.py:197:stop] 0/4370, RunningAvgSamplesPerSec=5.870011348349186, CurrSamplesPerSec=5.286158564834461, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2159/5000 [7:29:14<8:41:55, 11.02s/it][2022-12-20 17:51:50,270] [INFO] [timer.py:197:stop] 0/4372, RunningAvgSamplesPerSec=5.870009402718063, CurrSamplesPerSec=5.340187817688265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2160/5000 [7:29:25<8:41:53, 11.03s/it][2022-12-20 17:52:01,280] [INFO] [timer.py:197:stop] 0/4374, RunningAvgSamplesPerSec=5.87001347311125, CurrSamplesPerSec=5.333873591573434, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2161/5000 [7:29:36<8:41:29, 11.02s/it][2022-12-20 17:52:12,269] [INFO] [timer.py:197:stop] 0/4376, RunningAvgSamplesPerSec=5.8700222858667574, CurrSamplesPerSec=5.3644795497438, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2162/5000 [7:29:47<8:40:51, 11.01s/it][2022-12-20 17:52:23,275] [INFO] [timer.py:197:stop] 0/4378, RunningAvgSamplesPerSec=5.870027375099405, CurrSamplesPerSec=5.342871208550299, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2163/5000 [7:29:58<8:40:34, 11.01s/it][2022-12-20 17:52:34,313] [INFO] [logging.py:68:log_dist] [Rank 0] step=2190, skipped=3, lr=[6.253333333333333e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:52:34,314] [INFO] [timer.py:197:stop] 0/4380, RunningAvgSamplesPerSec=5.870023951950112, CurrSamplesPerSec=5.309192015665981, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2164/5000 [7:30:09<8:40:49, 11.02s/it][2022-12-20 17:52:45,355] [INFO] [timer.py:197:stop] 0/4382, RunningAvgSamplesPerSec=5.8700228460972586, CurrSamplesPerSec=5.300255263224404, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2165/5000 [7:30:20<8:40:56, 11.03s/it][2022-12-20 17:52:56,417] [INFO] [timer.py:197:stop] 0/4384, RunningAvgSamplesPerSec=5.870014232238921, CurrSamplesPerSec=5.304841376154955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2166/5000 [7:30:32<8:41:16, 11.04s/it][2022-12-20 17:53:07,469] [INFO] [timer.py:197:stop] 0/4386, RunningAvgSamplesPerSec=5.8700076871468205, CurrSamplesPerSec=5.312893139767952, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2167/5000 [7:30:43<8:41:19, 11.04s/it][2022-12-20 17:53:18,485] [INFO] [timer.py:197:stop] 0/4388, RunningAvgSamplesPerSec=5.870010227981493, CurrSamplesPerSec=5.301837258729139, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2168/5000 [7:30:54<8:40:46, 11.03s/it][2022-12-20 17:53:29,512] [INFO] [timer.py:197:stop] 0/4390, RunningAvgSamplesPerSec=5.870010098330011, CurrSamplesPerSec=5.337043586474473, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2169/5000 [7:31:05<8:40:30, 11.03s/it][2022-12-20 17:53:40,511] [INFO] [timer.py:197:stop] 0/4392, RunningAvgSamplesPerSec=5.870016919672246, CurrSamplesPerSec=5.308867354821755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2170/5000 [7:31:16<8:39:51, 11.02s/it][2022-12-20 17:53:51,528] [INFO] [timer.py:197:stop] 0/4394, RunningAvgSamplesPerSec=5.870018937985506, CurrSamplesPerSec=5.314870333896878, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2171/5000 [7:31:27<8:39:36, 11.02s/it][2022-12-20 17:54:02,543] [INFO] [timer.py:197:stop] 0/4396, RunningAvgSamplesPerSec=5.870021515999309, CurrSamplesPerSec=5.295017722947612, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2172/5000 [7:31:38<8:39:21, 11.02s/it][2022-12-20 17:54:13,563] [INFO] [timer.py:197:stop] 0/4398, RunningAvgSamplesPerSec=5.870023207670243, CurrSamplesPerSec=5.33698098154396, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2173/5000 [7:31:49<8:39:11, 11.02s/it][2022-12-20 17:54:23,695] [INFO] [logging.py:68:log_dist] [Rank 0] step=2200, skipped=3, lr=[6.231111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:54:23,697] [INFO] [timer.py:197:stop] 0/4400, RunningAvgSamplesPerSec=5.870239400872252, CurrSamplesPerSec=5.32010949416207, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 43%|████▎     | 2174/5000 [7:32:04<9:36:10, 12.23s/it][2022-12-20 17:54:34,701] [INFO] [timer.py:197:stop] 0/4402, RunningAvgSamplesPerSec=5.870244794134129, CurrSamplesPerSec=5.325983653056489, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2175/5000 [7:32:15<9:18:17, 11.86s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.22888888888889e-06, 'epoch': 53.05}

 44%|████▎     | 2175/5000 [7:32:15<9:18:17, 11.86s/it][2022-12-20 17:54:45,798] [INFO] [timer.py:197:stop] 0/4404, RunningAvgSamplesPerSec=5.870227706485792, CurrSamplesPerSec=5.226620921471748, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2176/5000 [7:32:26<9:08:35, 11.66s/it][2022-12-20 17:54:56,831] [INFO] [timer.py:197:stop] 0/4406, RunningAvgSamplesPerSec=5.8702261649024825, CurrSamplesPerSec=5.35868922853292, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2177/5000 [7:32:37<8:59:14, 11.46s/it][2022-12-20 17:55:07,875] [INFO] [timer.py:197:stop] 0/4408, RunningAvgSamplesPerSec=5.870221953911768, CurrSamplesPerSec=5.325334483644098, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2178/5000 [7:32:48<8:52:39, 11.33s/it][2022-12-20 17:55:18,908] [INFO] [timer.py:197:stop] 0/4410, RunningAvgSamplesPerSec=5.870220297986947, CurrSamplesPerSec=5.302666948437348, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2179/5000 [7:32:59<8:48:04, 11.23s/it][2022-12-20 17:55:29,917] [INFO] [timer.py:197:stop] 0/4412, RunningAvgSamplesPerSec=5.8702244810109026, CurrSamplesPerSec=5.306502896880908, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2180/5000 [7:33:10<8:44:46, 11.17s/it][2022-12-20 17:55:40,944] [INFO] [timer.py:197:stop] 0/4414, RunningAvgSamplesPerSec=5.870224377539624, CurrSamplesPerSec=5.2935157869495395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2181/5000 [7:33:21<8:43:20, 11.14s/it][2022-12-20 17:55:51,996] [INFO] [timer.py:197:stop] 0/4416, RunningAvgSamplesPerSec=5.870217996225358, CurrSamplesPerSec=5.314867808343779, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2182/5000 [7:33:32<8:42:11, 11.12s/it][2022-12-20 17:56:03,045] [INFO] [timer.py:197:stop] 0/4418, RunningAvgSamplesPerSec=5.870216377594884, CurrSamplesPerSec=5.333658450129131, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2183/5000 [7:33:43<8:40:40, 11.09s/it][2022-12-20 17:56:14,080] [INFO] [logging.py:68:log_dist] [Rank 0] step=2210, skipped=3, lr=[6.20888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:56:14,081] [INFO] [timer.py:197:stop] 0/4420, RunningAvgSamplesPerSec=5.870213945107222, CurrSamplesPerSec=5.3229291612897445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2184/5000 [7:33:54<8:39:52, 11.08s/it][2022-12-20 17:56:25,142] [INFO] [timer.py:197:stop] 0/4422, RunningAvgSamplesPerSec=5.87020532669836, CurrSamplesPerSec=5.309951322644149, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2185/5000 [7:34:05<8:38:14, 11.05s/it][2022-12-20 17:56:36,128] [INFO] [timer.py:197:stop] 0/4424, RunningAvgSamplesPerSec=5.870215394173533, CurrSamplesPerSec=5.300248146797007, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2186/5000 [7:34:16<8:38:30, 11.06s/it][2022-12-20 17:56:47,198] [INFO] [timer.py:197:stop] 0/4426, RunningAvgSamplesPerSec=5.8702046824193035, CurrSamplesPerSec=5.306991148024579, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▎     | 2187/5000 [7:34:27<8:37:52, 11.05s/it][2022-12-20 17:56:58,210] [INFO] [timer.py:197:stop] 0/4428, RunningAvgSamplesPerSec=5.870208108012052, CurrSamplesPerSec=5.316995384847527, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2188/5000 [7:34:38<8:37:12, 11.04s/it][2022-12-20 17:57:09,244] [INFO] [timer.py:197:stop] 0/4430, RunningAvgSamplesPerSec=5.870206100544846, CurrSamplesPerSec=5.297439279018229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2189/5000 [7:34:49<8:36:48, 11.03s/it][2022-12-20 17:57:20,201] [INFO] [timer.py:197:stop] 0/4432, RunningAvgSamplesPerSec=5.870223037162975, CurrSamplesPerSec=5.3535448829064745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2190/5000 [7:35:00<8:35:43, 11.01s/it][2022-12-20 17:57:31,178] [INFO] [timer.py:197:stop] 0/4434, RunningAvgSamplesPerSec=5.8702345546739, CurrSamplesPerSec=5.34425424013747, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2191/5000 [7:35:11<8:35:27, 11.01s/it][2022-12-20 17:57:42,179] [INFO] [timer.py:197:stop] 0/4436, RunningAvgSamplesPerSec=5.870240225084986, CurrSamplesPerSec=5.3483601073324305, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2192/5000 [7:35:22<8:34:15, 10.99s/it][2022-12-20 17:57:53,146] [INFO] [timer.py:197:stop] 0/4438, RunningAvgSamplesPerSec=5.870255408221334, CurrSamplesPerSec=5.324532960892702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2193/5000 [7:35:33<8:34:14, 10.99s/it][2022-12-20 17:58:04,109] [INFO] [logging.py:68:log_dist] [Rank 0] step=2220, skipped=3, lr=[6.186666666666668e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:58:04,111] [INFO] [timer.py:197:stop] 0/4440, RunningAvgSamplesPerSec=5.870270424616973, CurrSamplesPerSec=5.354673236326013, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2194/5000 [7:35:44<8:33:26, 10.98s/it][2022-12-20 17:58:15,076] [INFO] [timer.py:197:stop] 0/4442, RunningAvgSamplesPerSec=5.870285074820494, CurrSamplesPerSec=5.339975565876111, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2195/5000 [7:35:55<8:33:09, 10.98s/it][2022-12-20 17:58:26,004] [INFO] [timer.py:197:stop] 0/4444, RunningAvgSamplesPerSec=5.870308704588686, CurrSamplesPerSec=5.377953822621086, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2196/5000 [7:36:06<8:32:56, 10.98s/it][2022-12-20 17:58:37,065] [INFO] [timer.py:197:stop] 0/4446, RunningAvgSamplesPerSec=5.870304264304532, CurrSamplesPerSec=5.315643476426314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2197/5000 [7:36:17<8:33:29, 10.99s/it][2022-12-20 17:58:48,033] [INFO] [timer.py:197:stop] 0/4448, RunningAvgSamplesPerSec=5.870318229776591, CurrSamplesPerSec=5.354227647566181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2198/5000 [7:36:28<8:32:44, 10.98s/it][2022-12-20 17:58:59,035] [INFO] [timer.py:197:stop] 0/4450, RunningAvgSamplesPerSec=5.870323706572499, CurrSamplesPerSec=5.308686561565392, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2199/5000 [7:36:39<8:33:24, 11.00s/it][2022-12-20 17:59:10,047] [INFO] [timer.py:197:stop] 0/4452, RunningAvgSamplesPerSec=5.870327037311967, CurrSamplesPerSec=5.334483712637762, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2200/5000 [7:36:50<8:33:22, 11.00s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.173333333333333e-06, 'epoch': 53.65}

 44%|████▍     | 2200/5000 [7:36:50<8:33:22, 11.00s/it][2022-12-20 17:59:21,036] [INFO] [timer.py:197:stop] 0/4454, RunningAvgSamplesPerSec=5.870336345240717, CurrSamplesPerSec=5.353205594214484, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2201/5000 [7:37:01<8:33:09, 11.00s/it][2022-12-20 17:59:32,064] [INFO] [timer.py:197:stop] 0/4456, RunningAvgSamplesPerSec=5.870335691973032, CurrSamplesPerSec=5.324774828868988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2202/5000 [7:37:12<8:33:24, 11.01s/it][2022-12-20 17:59:43,110] [INFO] [timer.py:197:stop] 0/4458, RunningAvgSamplesPerSec=5.870330194346564, CurrSamplesPerSec=5.310666298899504, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2203/5000 [7:37:23<8:33:59, 11.03s/it][2022-12-20 17:59:54,192] [INFO] [logging.py:68:log_dist] [Rank 0] step=2230, skipped=3, lr=[6.1644444444444455e-06], mom=[[0.9, 0.999]]
[2022-12-20 17:59:54,194] [INFO] [timer.py:197:stop] 0/4460, RunningAvgSamplesPerSec=5.870315993061552, CurrSamplesPerSec=5.2943798390137315, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2204/5000 [7:37:34<8:34:26, 11.04s/it][2022-12-20 18:00:05,205] [INFO] [timer.py:197:stop] 0/4462, RunningAvgSamplesPerSec=5.87031917879999, CurrSamplesPerSec=5.347559309030412, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2205/5000 [7:37:45<8:33:33, 11.02s/it][2022-12-20 18:00:16,227] [INFO] [timer.py:197:stop] 0/4464, RunningAvgSamplesPerSec=5.870320164045288, CurrSamplesPerSec=5.319510246413381, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2206/5000 [7:37:56<8:33:21, 11.02s/it][2022-12-20 18:00:27,236] [INFO] [timer.py:197:stop] 0/4466, RunningAvgSamplesPerSec=5.870324146437861, CurrSamplesPerSec=5.331035968260933, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2207/5000 [7:38:07<8:32:58, 11.02s/it][2022-12-20 18:00:38,284] [INFO] [timer.py:197:stop] 0/4468, RunningAvgSamplesPerSec=5.870318796931328, CurrSamplesPerSec=5.298296664712766, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2208/5000 [7:38:18<8:33:16, 11.03s/it][2022-12-20 18:00:49,284] [INFO] [timer.py:197:stop] 0/4470, RunningAvgSamplesPerSec=5.870324975772341, CurrSamplesPerSec=5.346333005545231, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2209/5000 [7:38:29<8:32:43, 11.02s/it][2022-12-20 18:01:00,332] [INFO] [timer.py:197:stop] 0/4472, RunningAvgSamplesPerSec=5.870319495239341, CurrSamplesPerSec=5.306339886746351, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2210/5000 [7:38:40<8:32:52, 11.03s/it][2022-12-20 18:01:11,376] [INFO] [timer.py:197:stop] 0/4474, RunningAvgSamplesPerSec=5.87031502853921, CurrSamplesPerSec=5.309616066564138, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2211/5000 [7:38:51<8:32:50, 11.03s/it][2022-12-20 18:01:22,434] [INFO] [timer.py:197:stop] 0/4476, RunningAvgSamplesPerSec=5.870307104465001, CurrSamplesPerSec=5.29404383165413, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2212/5000 [7:39:02<8:32:55, 11.04s/it][2022-12-20 18:01:33,479] [INFO] [timer.py:197:stop] 0/4478, RunningAvgSamplesPerSec=5.870306279161913, CurrSamplesPerSec=5.299571545565143, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2213/5000 [7:39:14<8:32:37, 11.04s/it][2022-12-20 18:01:44,431] [INFO] [logging.py:68:log_dist] [Rank 0] step=2240, skipped=3, lr=[6.142222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:01:44,433] [INFO] [timer.py:197:stop] 0/4480, RunningAvgSamplesPerSec=5.87032324136822, CurrSamplesPerSec=5.366578373722497, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2214/5000 [7:39:24<8:31:01, 11.01s/it][2022-12-20 18:01:54,495] [INFO] [timer.py:197:stop] 0/4482, RunningAvgSamplesPerSec=5.870552127590959, CurrSamplesPerSec=6.273965263184023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 18:02:05,535] [INFO] [timer.py:197:stop] 0/4484, RunningAvgSamplesPerSec=5.870551357029141, CurrSamplesPerSec=5.322212989988344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2215/5000 [7:39:41<9:43:04, 12.56s/it][2022-12-20 18:02:16,518] [INFO] [timer.py:197:stop] 0/4486, RunningAvgSamplesPerSec=5.8705615437381935, CurrSamplesPerSec=5.318573691468843, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2216/5000 [7:39:52<9:20:53, 12.09s/it][2022-12-20 18:02:27,553] [INFO] [timer.py:197:stop] 0/4488, RunningAvgSamplesPerSec=5.870559359701557, CurrSamplesPerSec=5.308922162178412, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2217/5000 [7:40:03<9:06:02, 11.77s/it][2022-12-20 18:02:38,552] [INFO] [timer.py:197:stop] 0/4490, RunningAvgSamplesPerSec=5.870565793698353, CurrSamplesPerSec=5.349227447604921, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2218/5000 [7:40:14<8:55:04, 11.54s/it][2022-12-20 18:02:49,573] [INFO] [timer.py:197:stop] 0/4492, RunningAvgSamplesPerSec=5.870566941645717, CurrSamplesPerSec=5.338899269152701, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2219/5000 [7:40:25<8:47:41, 11.38s/it][2022-12-20 18:03:00,590] [INFO] [timer.py:197:stop] 0/4494, RunningAvgSamplesPerSec=5.87057296030001, CurrSamplesPerSec=5.30826664797813, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2220/5000 [7:40:36<8:42:21, 11.27s/it][2022-12-20 18:03:11,635] [INFO] [timer.py:197:stop] 0/4496, RunningAvgSamplesPerSec=5.870567957636121, CurrSamplesPerSec=5.302303704679044, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2221/5000 [7:40:47<8:39:00, 11.21s/it][2022-12-20 18:03:22,663] [INFO] [timer.py:197:stop] 0/4498, RunningAvgSamplesPerSec=5.8705674549640685, CurrSamplesPerSec=5.314936630524195, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2222/5000 [7:40:58<8:36:20, 11.15s/it][2022-12-20 18:03:33,662] [INFO] [logging.py:68:log_dist] [Rank 0] step=2250, skipped=3, lr=[6.120000000000001e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:03:33,664] [INFO] [timer.py:197:stop] 0/4500, RunningAvgSamplesPerSec=5.87057328942282, CurrSamplesPerSec=5.357323521758618, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2223/5000 [7:41:09<8:34:03, 11.11s/it][2022-12-20 18:03:44,671] [INFO] [timer.py:197:stop] 0/4502, RunningAvgSamplesPerSec=5.870577443053673, CurrSamplesPerSec=5.349235335744927, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2224/5000 [7:41:20<8:32:29, 11.08s/it][2022-12-20 18:03:55,730] [INFO] [timer.py:197:stop] 0/4504, RunningAvgSamplesPerSec=5.870569751496985, CurrSamplesPerSec=5.299186338843811, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 44%|████▍     | 2225/5000 [7:41:31<8:32:03, 11.07s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.1155555555555555e-06, 'epoch': 54.27}

 44%|████▍     | 2225/5000 [7:41:31<8:32:03, 11.07s/it][2022-12-20 18:04:06,826] [INFO] [timer.py:197:stop] 0/4506, RunningAvgSamplesPerSec=5.870553284526691, CurrSamplesPerSec=5.28072173446042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2226/5000 [7:41:42<8:32:12, 11.08s/it][2022-12-20 18:04:17,807] [INFO] [timer.py:197:stop] 0/4508, RunningAvgSamplesPerSec=5.870563717541872, CurrSamplesPerSec=5.341550753013409, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2227/5000 [7:41:53<8:30:40, 11.05s/it][2022-12-20 18:04:28,839] [INFO] [timer.py:197:stop] 0/4510, RunningAvgSamplesPerSec=5.870562004227285, CurrSamplesPerSec=5.312123950707151, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2228/5000 [7:42:04<8:30:14, 11.04s/it][2022-12-20 18:04:39,852] [INFO] [timer.py:197:stop] 0/4512, RunningAvgSamplesPerSec=5.87056507052006, CurrSamplesPerSec=5.3347385717229825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2229/5000 [7:42:15<8:29:37, 11.03s/it][2022-12-20 18:04:50,784] [INFO] [timer.py:197:stop] 0/4514, RunningAvgSamplesPerSec=5.870586964962464, CurrSamplesPerSec=5.34263088383506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2230/5000 [7:42:26<8:28:01, 11.00s/it][2022-12-20 18:05:01,863] [INFO] [timer.py:197:stop] 0/4516, RunningAvgSamplesPerSec=5.8705742430316725, CurrSamplesPerSec=5.2931830204812815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2231/5000 [7:42:37<8:28:52, 11.03s/it][2022-12-20 18:05:12,839] [INFO] [timer.py:197:stop] 0/4518, RunningAvgSamplesPerSec=5.87058590799604, CurrSamplesPerSec=5.328780144078012, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2232/5000 [7:42:48<8:27:59, 11.01s/it][2022-12-20 18:05:23,841] [INFO] [logging.py:68:log_dist] [Rank 0] step=2260, skipped=3, lr=[6.097777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:05:23,842] [INFO] [timer.py:197:stop] 0/4520, RunningAvgSamplesPerSec=5.870591104634569, CurrSamplesPerSec=5.342439064932352, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2233/5000 [7:42:59<8:27:41, 11.01s/it][2022-12-20 18:05:34,886] [INFO] [timer.py:197:stop] 0/4522, RunningAvgSamplesPerSec=5.870586754854745, CurrSamplesPerSec=5.304477624799513, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2234/5000 [7:43:10<8:27:59, 11.02s/it][2022-12-20 18:05:45,894] [INFO] [timer.py:197:stop] 0/4524, RunningAvgSamplesPerSec=5.87059072204096, CurrSamplesPerSec=5.3213124032765124, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2235/5000 [7:43:21<8:27:39, 11.02s/it][2022-12-20 18:05:56,953] [INFO] [timer.py:197:stop] 0/4526, RunningAvgSamplesPerSec=5.870582519940331, CurrSamplesPerSec=5.290731358429899, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2236/5000 [7:43:32<8:28:03, 11.03s/it][2022-12-20 18:06:07,944] [INFO] [timer.py:197:stop] 0/4528, RunningAvgSamplesPerSec=5.87059032690477, CurrSamplesPerSec=5.349775408863965, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2237/5000 [7:43:43<8:27:21, 11.02s/it][2022-12-20 18:06:18,924] [INFO] [timer.py:197:stop] 0/4530, RunningAvgSamplesPerSec=5.870601055139825, CurrSamplesPerSec=5.341148792232082, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2238/5000 [7:43:54<8:26:39, 11.01s/it][2022-12-20 18:06:29,916] [INFO] [timer.py:197:stop] 0/4532, RunningAvgSamplesPerSec=5.870609208945358, CurrSamplesPerSec=5.317973103961121, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2239/5000 [7:44:05<8:26:16, 11.00s/it][2022-12-20 18:06:40,900] [INFO] [timer.py:197:stop] 0/4534, RunningAvgSamplesPerSec=5.870618986275708, CurrSamplesPerSec=5.323115781631799, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2240/5000 [7:44:16<8:25:50, 11.00s/it][2022-12-20 18:06:51,890] [INFO] [timer.py:197:stop] 0/4536, RunningAvgSamplesPerSec=5.870627128086081, CurrSamplesPerSec=5.3152138314802375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2241/5000 [7:44:27<8:25:34, 10.99s/it][2022-12-20 18:07:02,851] [INFO] [timer.py:197:stop] 0/4538, RunningAvgSamplesPerSec=5.870642433326336, CurrSamplesPerSec=5.347074216072864, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2242/5000 [7:44:38<8:24:55, 10.98s/it][2022-12-20 18:07:13,865] [INFO] [logging.py:68:log_dist] [Rank 0] step=2270, skipped=3, lr=[6.075555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:07:13,868] [INFO] [timer.py:197:stop] 0/4540, RunningAvgSamplesPerSec=5.8706444135745235, CurrSamplesPerSec=5.313394346954689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2243/5000 [7:44:49<8:25:10, 10.99s/it][2022-12-20 18:07:24,926] [INFO] [timer.py:197:stop] 0/4542, RunningAvgSamplesPerSec=5.8706363597859665, CurrSamplesPerSec=5.328197978259616, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2244/5000 [7:45:00<8:25:53, 11.01s/it][2022-12-20 18:07:35,924] [INFO] [timer.py:197:stop] 0/4544, RunningAvgSamplesPerSec=5.870642901433451, CurrSamplesPerSec=5.344200190390852, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2245/5000 [7:45:11<8:25:29, 11.01s/it][2022-12-20 18:07:46,929] [INFO] [timer.py:197:stop] 0/4546, RunningAvgSamplesPerSec=5.870647615742826, CurrSamplesPerSec=5.340968556594532, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2246/5000 [7:45:22<8:25:15, 11.01s/it][2022-12-20 18:07:57,866] [INFO] [timer.py:197:stop] 0/4548, RunningAvgSamplesPerSec=5.870668448988069, CurrSamplesPerSec=5.343126444309703, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2247/5000 [7:45:33<8:24:05, 10.99s/it][2022-12-20 18:08:08,904] [INFO] [timer.py:197:stop] 0/4550, RunningAvgSamplesPerSec=5.870665197017616, CurrSamplesPerSec=5.313265197662393, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2248/5000 [7:45:44<8:24:37, 11.00s/it][2022-12-20 18:08:19,925] [INFO] [timer.py:197:stop] 0/4552, RunningAvgSamplesPerSec=5.870666064435402, CurrSamplesPerSec=5.321725099182347, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▍     | 2249/5000 [7:45:55<8:24:42, 11.01s/it][2022-12-20 18:08:30,912] [INFO] [timer.py:197:stop] 0/4554, RunningAvgSamplesPerSec=5.870675043040991, CurrSamplesPerSec=5.33792679081686, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2250/5000 [7:46:06<8:24:14, 11.00s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.0600000000000004e-06, 'epoch': 54.87}

 45%|████▌     | 2250/5000 [7:46:06<8:24:14, 11.00s/it][2022-12-20 18:08:41,862] [INFO] [timer.py:197:stop] 0/4556, RunningAvgSamplesPerSec=5.870693082485557, CurrSamplesPerSec=5.3985375515628995, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2251/5000 [7:46:17<8:23:20, 10.99s/it][2022-12-20 18:08:52,903] [INFO] [timer.py:197:stop] 0/4558, RunningAvgSamplesPerSec=5.8706893069688215, CurrSamplesPerSec=5.310890096822702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2252/5000 [7:46:28<8:23:54, 11.00s/it][2022-12-20 18:09:03,916] [INFO] [logging.py:68:log_dist] [Rank 0] step=2280, skipped=3, lr=[6.0533333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:09:03,917] [INFO] [timer.py:197:stop] 0/4560, RunningAvgSamplesPerSec=5.8706915175928085, CurrSamplesPerSec=5.307560713025237, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2253/5000 [7:46:39<8:23:53, 11.01s/it][2022-12-20 18:09:14,968] [INFO] [timer.py:197:stop] 0/4562, RunningAvgSamplesPerSec=5.870685772162035, CurrSamplesPerSec=5.310349231600586, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2254/5000 [7:46:50<8:24:18, 11.02s/it][2022-12-20 18:09:26,012] [INFO] [timer.py:197:stop] 0/4564, RunningAvgSamplesPerSec=5.870681150043331, CurrSamplesPerSec=5.317140303056774, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2255/5000 [7:47:01<8:24:28, 11.03s/it][2022-12-20 18:09:36,100] [INFO] [timer.py:197:stop] 0/4566, RunningAvgSamplesPerSec=5.870899919603606, CurrSamplesPerSec=5.3357597364753175, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2256/5000 [7:47:16<9:19:15, 12.23s/it][2022-12-20 18:09:47,143] [INFO] [timer.py:197:stop] 0/4568, RunningAvgSamplesPerSec=5.870895442935629, CurrSamplesPerSec=5.302241074251752, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2257/5000 [7:47:27<9:02:57, 11.88s/it][2022-12-20 18:09:58,184] [INFO] [timer.py:197:stop] 0/4570, RunningAvgSamplesPerSec=5.87089147654617, CurrSamplesPerSec=5.315417804178863, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2258/5000 [7:47:38<8:51:34, 11.63s/it][2022-12-20 18:10:09,202] [INFO] [timer.py:197:stop] 0/4572, RunningAvgSamplesPerSec=5.870892897873652, CurrSamplesPerSec=5.353499826991517, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2259/5000 [7:47:49<8:42:36, 11.44s/it][2022-12-20 18:10:20,207] [INFO] [timer.py:197:stop] 0/4574, RunningAvgSamplesPerSec=5.870899033808796, CurrSamplesPerSec=5.341302894846542, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2260/5000 [7:48:00<8:35:46, 11.29s/it][2022-12-20 18:10:31,130] [INFO] [timer.py:197:stop] 0/4576, RunningAvgSamplesPerSec=5.870922645187618, CurrSamplesPerSec=5.369947644724779, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2261/5000 [7:48:11<8:30:45, 11.19s/it][2022-12-20 18:10:42,116] [INFO] [timer.py:197:stop] 0/4578, RunningAvgSamplesPerSec=5.8709316067586546, CurrSamplesPerSec=5.330570381480216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2262/5000 [7:48:22<8:28:10, 11.14s/it][2022-12-20 18:10:53,131] [INFO] [logging.py:68:log_dist] [Rank 0] step=2290, skipped=3, lr=[6.031111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:10:53,133] [INFO] [timer.py:197:stop] 0/4580, RunningAvgSamplesPerSec=5.870933116073864, CurrSamplesPerSec=5.327448457889522, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2263/5000 [7:48:33<8:25:20, 11.08s/it][2022-12-20 18:11:04,100] [INFO] [timer.py:197:stop] 0/4582, RunningAvgSamplesPerSec=5.8709464092326655, CurrSamplesPerSec=5.305196999188752, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2264/5000 [7:48:44<8:24:02, 11.05s/it][2022-12-20 18:11:15,085] [INFO] [timer.py:197:stop] 0/4584, RunningAvgSamplesPerSec=5.870955350393718, CurrSamplesPerSec=5.315038919993688, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2265/5000 [7:48:55<8:23:57, 11.06s/it][2022-12-20 18:11:26,125] [INFO] [timer.py:197:stop] 0/4586, RunningAvgSamplesPerSec=5.870951370397516, CurrSamplesPerSec=5.334038085803798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2266/5000 [7:49:06<8:23:09, 11.04s/it][2022-12-20 18:11:37,186] [INFO] [timer.py:197:stop] 0/4588, RunningAvgSamplesPerSec=5.870942316040723, CurrSamplesPerSec=5.2892047560615465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2267/5000 [7:49:17<8:23:26, 11.05s/it][2022-12-20 18:11:48,232] [INFO] [timer.py:197:stop] 0/4590, RunningAvgSamplesPerSec=5.870937003251864, CurrSamplesPerSec=5.315885169074902, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2268/5000 [7:49:28<8:22:38, 11.04s/it][2022-12-20 18:11:59,202] [INFO] [timer.py:197:stop] 0/4592, RunningAvgSamplesPerSec=5.870949679704978, CurrSamplesPerSec=5.350317723843352, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2269/5000 [7:49:39<8:21:54, 11.03s/it][2022-12-20 18:12:10,222] [INFO] [timer.py:197:stop] 0/4594, RunningAvgSamplesPerSec=5.870950831891253, CurrSamplesPerSec=5.33158402088243, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2270/5000 [7:49:50<8:21:14, 11.02s/it][2022-12-20 18:12:21,218] [INFO] [timer.py:197:stop] 0/4596, RunningAvgSamplesPerSec=5.870957431204611, CurrSamplesPerSec=5.327416950506657, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2271/5000 [7:50:01<8:21:17, 11.02s/it][2022-12-20 18:12:32,202] [INFO] [timer.py:197:stop] 0/4598, RunningAvgSamplesPerSec=5.870966682666349, CurrSamplesPerSec=5.371038217012425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2272/5000 [7:50:12<8:20:35, 11.01s/it][2022-12-20 18:12:43,263] [INFO] [logging.py:68:log_dist] [Rank 0] step=2300, skipped=3, lr=[6.00888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:12:43,265] [INFO] [timer.py:197:stop] 0/4600, RunningAvgSamplesPerSec=5.870957457904608, CurrSamplesPerSec=5.300088660113958, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2273/5000 [7:50:23<8:21:01, 11.02s/it][2022-12-20 18:12:54,321] [INFO] [timer.py:197:stop] 0/4602, RunningAvgSamplesPerSec=5.870950026559844, CurrSamplesPerSec=5.301217203297933, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 45%|████▌     | 2274/5000 [7:50:34<8:20:58, 11.03s/it][2022-12-20 18:13:05,333] [INFO] [timer.py:197:stop] 0/4604, RunningAvgSamplesPerSec=5.870952996175184, CurrSamplesPerSec=5.320126786213654, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2275/5000 [7:50:45<8:20:25, 11.02s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 6.004444444444445e-06, 'epoch': 55.48}

 46%|████▌     | 2275/5000 [7:50:45<8:20:25, 11.02s/it][2022-12-20 18:13:16,356] [INFO] [timer.py:197:stop] 0/4606, RunningAvgSamplesPerSec=5.870954273438248, CurrSamplesPerSec=5.302377648191757, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2276/5000 [7:50:56<8:20:08, 11.02s/it][2022-12-20 18:13:27,415] [INFO] [timer.py:197:stop] 0/4608, RunningAvgSamplesPerSec=5.870947788452088, CurrSamplesPerSec=5.262222986141363, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2277/5000 [7:51:07<8:21:24, 11.05s/it][2022-12-20 18:13:38,499] [INFO] [timer.py:197:stop] 0/4610, RunningAvgSamplesPerSec=5.870933892110724, CurrSamplesPerSec=5.291357307374993, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2278/5000 [7:51:19<8:21:30, 11.05s/it][2022-12-20 18:13:49,553] [INFO] [timer.py:197:stop] 0/4612, RunningAvgSamplesPerSec=5.870927525029095, CurrSamplesPerSec=5.304561482358754, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2279/5000 [7:51:30<8:20:59, 11.05s/it][2022-12-20 18:14:00,542] [INFO] [timer.py:197:stop] 0/4614, RunningAvgSamplesPerSec=5.870937168649814, CurrSamplesPerSec=5.341187689018548, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2280/5000 [7:51:41<8:20:28, 11.04s/it][2022-12-20 18:14:11,614] [INFO] [timer.py:197:stop] 0/4616, RunningAvgSamplesPerSec=5.8709257256057805, CurrSamplesPerSec=5.296230418571072, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2281/5000 [7:51:52<8:20:15, 11.04s/it][2022-12-20 18:14:22,644] [INFO] [timer.py:197:stop] 0/4618, RunningAvgSamplesPerSec=5.870924313676961, CurrSamplesPerSec=5.30355369006467, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2282/5000 [7:52:03<8:19:56, 11.04s/it][2022-12-20 18:14:33,697] [INFO] [logging.py:68:log_dist] [Rank 0] step=2310, skipped=3, lr=[5.986666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:14:33,698] [INFO] [timer.py:197:stop] 0/4620, RunningAvgSamplesPerSec=5.870917361602176, CurrSamplesPerSec=5.282387928496305, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2283/5000 [7:52:14<8:20:29, 11.05s/it][2022-12-20 18:14:44,751] [INFO] [timer.py:197:stop] 0/4622, RunningAvgSamplesPerSec=5.870912329417045, CurrSamplesPerSec=5.315249194046155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2284/5000 [7:52:25<8:19:52, 11.04s/it][2022-12-20 18:14:55,752] [INFO] [timer.py:197:stop] 0/4624, RunningAvgSamplesPerSec=5.870918348324344, CurrSamplesPerSec=5.334660966461219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2285/5000 [7:52:36<8:18:52, 11.03s/it][2022-12-20 18:15:06,771] [INFO] [timer.py:197:stop] 0/4626, RunningAvgSamplesPerSec=5.870919689238294, CurrSamplesPerSec=5.301260336536591, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2286/5000 [7:52:47<8:19:08, 11.03s/it][2022-12-20 18:15:17,787] [INFO] [timer.py:197:stop] 0/4628, RunningAvgSamplesPerSec=5.870921556314186, CurrSamplesPerSec=5.338307034374673, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2287/5000 [7:52:58<8:18:48, 11.03s/it][2022-12-20 18:15:28,918] [INFO] [timer.py:197:stop] 0/4630, RunningAvgSamplesPerSec=5.870896769145393, CurrSamplesPerSec=5.243830854016576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2288/5000 [7:53:09<8:20:05, 11.06s/it][2022-12-20 18:15:39,944] [INFO] [timer.py:197:stop] 0/4632, RunningAvgSamplesPerSec=5.8708967228023345, CurrSamplesPerSec=5.343863574451488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2289/5000 [7:53:20<8:19:16, 11.05s/it][2022-12-20 18:15:50,982] [INFO] [timer.py:197:stop] 0/4634, RunningAvgSamplesPerSec=5.870893957067914, CurrSamplesPerSec=5.325957023854675, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2290/5000 [7:53:31<8:17:47, 11.02s/it][2022-12-20 18:16:01,965] [INFO] [timer.py:197:stop] 0/4636, RunningAvgSamplesPerSec=5.870903716027733, CurrSamplesPerSec=5.299895698245635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2291/5000 [7:53:42<8:18:01, 11.03s/it][2022-12-20 18:16:13,023] [INFO] [timer.py:197:stop] 0/4638, RunningAvgSamplesPerSec=5.8708973809534974, CurrSamplesPerSec=5.300044290197461, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2292/5000 [7:53:53<8:17:39, 11.03s/it][2022-12-20 18:16:24,058] [INFO] [logging.py:68:log_dist] [Rank 0] step=2320, skipped=3, lr=[5.964444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:16:24,059] [INFO] [timer.py:197:stop] 0/4640, RunningAvgSamplesPerSec=5.870894541327961, CurrSamplesPerSec=5.2769881011772455, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2293/5000 [7:54:04<8:18:23, 11.05s/it][2022-12-20 18:16:35,116] [INFO] [timer.py:197:stop] 0/4642, RunningAvgSamplesPerSec=5.8708870822757655, CurrSamplesPerSec=5.310068546224846, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2294/5000 [7:54:15<8:18:07, 11.05s/it][2022-12-20 18:16:46,120] [INFO] [timer.py:197:stop] 0/4644, RunningAvgSamplesPerSec=5.870892285797219, CurrSamplesPerSec=5.34450301074386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2295/5000 [7:54:26<8:17:21, 11.03s/it][2022-12-20 18:16:57,142] [INFO] [timer.py:197:stop] 0/4646, RunningAvgSamplesPerSec=5.870894343718299, CurrSamplesPerSec=5.327156658802541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2296/5000 [7:54:37<8:17:01, 11.03s/it][2022-12-20 18:17:07,242] [INFO] [timer.py:197:stop] 0/4648, RunningAvgSamplesPerSec=5.87110609209428, CurrSamplesPerSec=6.275430511545547, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 18:17:18,222] [INFO] [timer.py:197:stop] 0/4650, RunningAvgSamplesPerSec=5.8711192627637, CurrSamplesPerSec=5.344714984127457, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2297/5000 [7:54:53<9:25:49, 12.56s/it][2022-12-20 18:17:29,263] [INFO] [timer.py:197:stop] 0/4652, RunningAvgSamplesPerSec=5.871115269475744, CurrSamplesPerSec=5.303019975204742, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2298/5000 [7:55:04<9:05:05, 12.10s/it][2022-12-20 18:17:40,278] [INFO] [timer.py:197:stop] 0/4654, RunningAvgSamplesPerSec=5.8711173808462425, CurrSamplesPerSec=5.343142822770184, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2299/5000 [7:55:15<8:50:10, 11.78s/it][2022-12-20 18:17:51,404] [INFO] [timer.py:197:stop] 0/4656, RunningAvgSamplesPerSec=5.871097705050236, CurrSamplesPerSec=5.299646040670556, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2300/5000 [7:55:26<8:41:11, 11.58s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.946666666666668e-06, 'epoch': 56.1}

 46%|████▌     | 2300/5000 [7:55:27<8:41:11, 11.58s/it][2022-12-20 18:18:02,413] [INFO] [timer.py:197:stop] 0/4658, RunningAvgSamplesPerSec=5.871103440017517, CurrSamplesPerSec=5.326089115764955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2301/5000 [7:55:38<8:33:16, 11.41s/it][2022-12-20 18:18:13,424] [INFO] [logging.py:68:log_dist] [Rank 0] step=2330, skipped=3, lr=[5.9422222222222225e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:18:13,426] [INFO] [timer.py:197:stop] 0/4660, RunningAvgSamplesPerSec=5.871106384133189, CurrSamplesPerSec=5.327966585756051, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2302/5000 [7:55:49<8:27:42, 11.29s/it][2022-12-20 18:18:24,600] [INFO] [timer.py:197:stop] 0/4662, RunningAvgSamplesPerSec=5.871076016291257, CurrSamplesPerSec=5.295388743821942, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2303/5000 [7:56:00<8:25:57, 11.26s/it][2022-12-20 18:18:35,585] [INFO] [timer.py:197:stop] 0/4664, RunningAvgSamplesPerSec=5.8710852238829885, CurrSamplesPerSec=5.352689590593273, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2304/5000 [7:56:11<8:22:06, 11.17s/it][2022-12-20 18:18:46,618] [INFO] [timer.py:197:stop] 0/4666, RunningAvgSamplesPerSec=5.871083356925433, CurrSamplesPerSec=5.310553881707882, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2305/5000 [7:56:22<8:20:01, 11.13s/it][2022-12-20 18:18:57,552] [INFO] [timer.py:197:stop] 0/4668, RunningAvgSamplesPerSec=5.871104311904402, CurrSamplesPerSec=5.380907637614472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2306/5000 [7:56:33<8:17:09, 11.07s/it][2022-12-20 18:19:08,457] [INFO] [timer.py:197:stop] 0/4670, RunningAvgSamplesPerSec=5.871131886742229, CurrSamplesPerSec=5.364129440848581, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2307/5000 [7:56:44<8:14:43, 11.02s/it][2022-12-20 18:19:19,525] [INFO] [timer.py:197:stop] 0/4672, RunningAvgSamplesPerSec=5.871121644112818, CurrSamplesPerSec=5.315448327755886, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2308/5000 [7:56:55<8:15:09, 11.04s/it][2022-12-20 18:19:30,518] [INFO] [timer.py:197:stop] 0/4674, RunningAvgSamplesPerSec=5.8711287193253865, CurrSamplesPerSec=5.347486656655464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2309/5000 [7:57:06<8:14:23, 11.02s/it][2022-12-20 18:19:41,518] [INFO] [timer.py:197:stop] 0/4676, RunningAvgSamplesPerSec=5.871134100577123, CurrSamplesPerSec=5.351351049627345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2310/5000 [7:57:17<8:13:53, 11.02s/it][2022-12-20 18:19:52,568] [INFO] [timer.py:197:stop] 0/4678, RunningAvgSamplesPerSec=5.871128114351411, CurrSamplesPerSec=5.321487094957284, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2311/5000 [7:57:28<8:14:09, 11.03s/it][2022-12-20 18:20:03,554] [INFO] [logging.py:68:log_dist] [Rank 0] step=2340, skipped=3, lr=[5.92e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:20:03,555] [INFO] [timer.py:197:stop] 0/4680, RunningAvgSamplesPerSec=5.8711365683614485, CurrSamplesPerSec=5.312783782743919, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▌     | 2312/5000 [7:57:39<8:13:27, 11.01s/it][2022-12-20 18:20:14,533] [INFO] [timer.py:197:stop] 0/4682, RunningAvgSamplesPerSec=5.871146973454374, CurrSamplesPerSec=5.3725575904531935, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2313/5000 [7:57:50<8:12:47, 11.00s/it][2022-12-20 18:20:25,521] [INFO] [timer.py:197:stop] 0/4684, RunningAvgSamplesPerSec=5.871156271139281, CurrSamplesPerSec=5.351185912563284, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2314/5000 [7:58:01<8:12:22, 11.00s/it][2022-12-20 18:20:36,520] [INFO] [timer.py:197:stop] 0/4686, RunningAvgSamplesPerSec=5.871162687798975, CurrSamplesPerSec=5.348481803636464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2315/5000 [7:58:12<8:12:11, 11.00s/it][2022-12-20 18:20:47,538] [INFO] [timer.py:197:stop] 0/4688, RunningAvgSamplesPerSec=5.8711639430197415, CurrSamplesPerSec=5.329260653619353, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2316/5000 [7:58:23<8:12:16, 11.00s/it][2022-12-20 18:20:58,593] [INFO] [timer.py:197:stop] 0/4690, RunningAvgSamplesPerSec=5.871156808925758, CurrSamplesPerSec=5.307966030657009, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2317/5000 [7:58:34<8:12:46, 11.02s/it][2022-12-20 18:21:09,564] [INFO] [timer.py:197:stop] 0/4692, RunningAvgSamplesPerSec=5.871169346033693, CurrSamplesPerSec=5.333274206181227, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2318/5000 [7:58:45<8:11:56, 11.01s/it][2022-12-20 18:21:20,614] [INFO] [timer.py:197:stop] 0/4694, RunningAvgSamplesPerSec=5.871164018137679, CurrSamplesPerSec=5.305311286760566, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2319/5000 [7:58:56<8:12:20, 11.02s/it][2022-12-20 18:21:31,594] [INFO] [timer.py:197:stop] 0/4696, RunningAvgSamplesPerSec=5.871174517765798, CurrSamplesPerSec=5.368504257877519, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2320/5000 [7:59:07<8:11:38, 11.01s/it][2022-12-20 18:21:42,598] [INFO] [timer.py:197:stop] 0/4698, RunningAvgSamplesPerSec=5.871179260390596, CurrSamplesPerSec=5.351455812402581, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2321/5000 [7:59:18<8:11:25, 11.01s/it][2022-12-20 18:21:53,552] [INFO] [logging.py:68:log_dist] [Rank 0] step=2350, skipped=3, lr=[5.897777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:21:53,553] [INFO] [timer.py:197:stop] 0/4700, RunningAvgSamplesPerSec=5.871195088016474, CurrSamplesPerSec=5.311177804678387, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2322/5000 [7:59:29<8:10:33, 10.99s/it][2022-12-20 18:22:04,587] [INFO] [timer.py:197:stop] 0/4702, RunningAvgSamplesPerSec=5.871192897721099, CurrSamplesPerSec=5.311430021278926, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2323/5000 [7:59:40<8:10:57, 11.00s/it][2022-12-20 18:22:15,591] [INFO] [timer.py:197:stop] 0/4704, RunningAvgSamplesPerSec=5.871197949589899, CurrSamplesPerSec=5.310524254791887, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2324/5000 [7:59:51<8:10:45, 11.00s/it][2022-12-20 18:22:26,617] [INFO] [timer.py:197:stop] 0/4706, RunningAvgSamplesPerSec=5.871198485794172, CurrSamplesPerSec=5.337180898024717, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 46%|████▋     | 2325/5000 [8:00:02<8:10:53, 11.01s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.891111111111112e-06, 'epoch': 56.7}

 46%|████▋     | 2325/5000 [8:00:02<8:10:53, 11.01s/it][2022-12-20 18:22:37,646] [INFO] [timer.py:197:stop] 0/4708, RunningAvgSamplesPerSec=5.8711978476400235, CurrSamplesPerSec=5.302749072724235, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2326/5000 [8:00:13<8:10:57, 11.02s/it][2022-12-20 18:22:48,684] [INFO] [timer.py:197:stop] 0/4710, RunningAvgSamplesPerSec=5.871195166364643, CurrSamplesPerSec=5.297592751576862, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2327/5000 [8:00:24<8:11:03, 11.02s/it][2022-12-20 18:22:59,692] [INFO] [timer.py:197:stop] 0/4712, RunningAvgSamplesPerSec=5.871202369437678, CurrSamplesPerSec=5.334848410351626, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2328/5000 [8:00:35<8:10:40, 11.02s/it][2022-12-20 18:23:10,747] [INFO] [timer.py:197:stop] 0/4714, RunningAvgSamplesPerSec=5.871195319756936, CurrSamplesPerSec=5.31674285004424, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2329/5000 [8:00:46<8:10:59, 11.03s/it][2022-12-20 18:23:21,764] [INFO] [timer.py:197:stop] 0/4716, RunningAvgSamplesPerSec=5.871197301076524, CurrSamplesPerSec=5.346963660428676, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2330/5000 [8:00:57<8:10:37, 11.03s/it][2022-12-20 18:23:32,838] [INFO] [timer.py:197:stop] 0/4718, RunningAvgSamplesPerSec=5.871186023619932, CurrSamplesPerSec=5.286369891301856, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2331/5000 [8:01:08<8:11:06, 11.04s/it][2022-12-20 18:23:44,171] [INFO] [logging.py:68:log_dist] [Rank 0] step=2360, skipped=3, lr=[5.875555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:23:44,173] [INFO] [timer.py:197:stop] 0/4720, RunningAvgSamplesPerSec=5.871114862906311, CurrSamplesPerSec=5.04660795426858, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2332/5000 [8:01:19<8:14:50, 11.13s/it][2022-12-20 18:23:55,241] [INFO] [timer.py:197:stop] 0/4722, RunningAvgSamplesPerSec=5.87110466036547, CurrSamplesPerSec=5.296698177726609, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2333/5000 [8:01:30<8:13:51, 11.11s/it][2022-12-20 18:24:06,270] [INFO] [timer.py:197:stop] 0/4724, RunningAvgSamplesPerSec=5.87110379322603, CurrSamplesPerSec=5.336869994039957, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2334/5000 [8:01:41<8:12:35, 11.09s/it][2022-12-20 18:24:17,310] [INFO] [timer.py:197:stop] 0/4726, RunningAvgSamplesPerSec=5.871100236188354, CurrSamplesPerSec=5.294814895116352, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2335/5000 [8:01:52<8:11:47, 11.07s/it][2022-12-20 18:24:28,332] [INFO] [timer.py:197:stop] 0/4728, RunningAvgSamplesPerSec=5.871100753572674, CurrSamplesPerSec=5.295850503445754, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2336/5000 [8:02:03<8:10:55, 11.06s/it][2022-12-20 18:24:39,407] [INFO] [timer.py:197:stop] 0/4730, RunningAvgSamplesPerSec=5.871089200046865, CurrSamplesPerSec=5.291451807036544, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2337/5000 [8:02:15<8:10:59, 11.06s/it][2022-12-20 18:24:49,459] [INFO] [timer.py:197:stop] 0/4732, RunningAvgSamplesPerSec=5.871308701399774, CurrSamplesPerSec=5.382618451994167, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2338/5000 [8:02:30<9:03:16, 12.25s/it][2022-12-20 18:25:00,466] [INFO] [timer.py:197:stop] 0/4734, RunningAvgSamplesPerSec=5.871312538646093, CurrSamplesPerSec=5.341324151077426, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2339/5000 [8:02:40<8:46:08, 11.86s/it][2022-12-20 18:25:11,474] [INFO] [timer.py:197:stop] 0/4736, RunningAvgSamplesPerSec=5.871316042301735, CurrSamplesPerSec=5.310430543461063, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2340/5000 [8:02:52<8:35:37, 11.63s/it][2022-12-20 18:25:22,572] [INFO] [timer.py:197:stop] 0/4738, RunningAvgSamplesPerSec=5.871308931966556, CurrSamplesPerSec=5.301230394456609, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2341/5000 [8:03:03<8:27:41, 11.46s/it][2022-12-20 18:25:33,647] [INFO] [logging.py:68:log_dist] [Rank 0] step=2370, skipped=3, lr=[5.853333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:25:33,648] [INFO] [timer.py:197:stop] 0/4740, RunningAvgSamplesPerSec=5.871296781156275, CurrSamplesPerSec=5.276104412658258, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2342/5000 [8:03:14<8:21:57, 11.33s/it][2022-12-20 18:25:44,669] [INFO] [timer.py:197:stop] 0/4742, RunningAvgSamplesPerSec=5.8712971890803205, CurrSamplesPerSec=5.291811687642197, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2343/5000 [8:03:25<8:17:39, 11.24s/it][2022-12-20 18:25:55,642] [INFO] [timer.py:197:stop] 0/4744, RunningAvgSamplesPerSec=5.871308739807625, CurrSamplesPerSec=5.335759948595829, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2344/5000 [8:03:36<8:14:24, 11.17s/it][2022-12-20 18:26:06,699] [INFO] [timer.py:197:stop] 0/4746, RunningAvgSamplesPerSec=5.871300974582938, CurrSamplesPerSec=5.291530663642193, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2345/5000 [8:03:47<8:12:54, 11.14s/it][2022-12-20 18:26:17,772] [INFO] [timer.py:197:stop] 0/4748, RunningAvgSamplesPerSec=5.871289917901719, CurrSamplesPerSec=5.289992758160313, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2346/5000 [8:03:58<8:11:56, 11.12s/it][2022-12-20 18:26:28,780] [INFO] [timer.py:197:stop] 0/4750, RunningAvgSamplesPerSec=5.871293634939237, CurrSamplesPerSec=5.354359650311551, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2347/5000 [8:04:09<8:10:13, 11.09s/it][2022-12-20 18:26:39,852] [INFO] [timer.py:197:stop] 0/4752, RunningAvgSamplesPerSec=5.871287929908939, CurrSamplesPerSec=5.3035438404076505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2348/5000 [8:04:20<8:10:13, 11.09s/it][2022-12-20 18:26:50,952] [INFO] [timer.py:197:stop] 0/4754, RunningAvgSamplesPerSec=5.871278136640009, CurrSamplesPerSec=5.309591491186154, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2349/5000 [8:04:31<8:09:55, 11.09s/it][2022-12-20 18:27:01,952] [INFO] [timer.py:197:stop] 0/4756, RunningAvgSamplesPerSec=5.871284974427918, CurrSamplesPerSec=5.368724367177203, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2350/5000 [8:04:42<8:09:02, 11.07s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.8355555555555565e-06, 'epoch': 57.31}

 47%|████▋     | 2350/5000 [8:04:42<8:09:02, 11.07s/it][2022-12-20 18:27:13,046] [INFO] [timer.py:197:stop] 0/4758, RunningAvgSamplesPerSec=5.8712767712992, CurrSamplesPerSec=5.331145654472759, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2351/5000 [8:04:53<8:08:48, 11.07s/it][2022-12-20 18:27:24,113] [INFO] [logging.py:68:log_dist] [Rank 0] step=2380, skipped=3, lr=[5.831111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:27:24,114] [INFO] [timer.py:197:stop] 0/4760, RunningAvgSamplesPerSec=5.871270703559603, CurrSamplesPerSec=5.3177923220746, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2352/5000 [8:05:04<8:07:54, 11.06s/it][2022-12-20 18:27:35,086] [INFO] [timer.py:197:stop] 0/4762, RunningAvgSamplesPerSec=5.871286168924748, CurrSamplesPerSec=5.357933458545512, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2353/5000 [8:05:15<8:06:22, 11.02s/it][2022-12-20 18:27:46,025] [INFO] [timer.py:197:stop] 0/4764, RunningAvgSamplesPerSec=5.871308360278715, CurrSamplesPerSec=5.37100146342459, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2354/5000 [8:05:26<8:05:33, 11.01s/it][2022-12-20 18:27:57,057] [INFO] [timer.py:197:stop] 0/4766, RunningAvgSamplesPerSec=5.871306616009069, CurrSamplesPerSec=5.322218055067927, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2355/5000 [8:05:37<8:05:31, 11.01s/it][2022-12-20 18:28:08,102] [INFO] [timer.py:197:stop] 0/4768, RunningAvgSamplesPerSec=5.871301614768655, CurrSamplesPerSec=5.300375408148057, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2356/5000 [8:05:48<8:05:53, 11.03s/it][2022-12-20 18:28:19,056] [INFO] [timer.py:197:stop] 0/4770, RunningAvgSamplesPerSec=5.87131752311646, CurrSamplesPerSec=5.3925441652102295, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2357/5000 [8:05:59<8:04:19, 10.99s/it][2022-12-20 18:28:30,022] [INFO] [timer.py:197:stop] 0/4772, RunningAvgSamplesPerSec=5.871330422922975, CurrSamplesPerSec=5.3519854499454285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2358/5000 [8:06:10<8:03:45, 10.99s/it][2022-12-20 18:28:40,993] [INFO] [timer.py:197:stop] 0/4774, RunningAvgSamplesPerSec=5.871342457862554, CurrSamplesPerSec=5.347253159751435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2359/5000 [8:06:21<8:03:25, 10.98s/it][2022-12-20 18:28:51,986] [INFO] [timer.py:197:stop] 0/4776, RunningAvgSamplesPerSec=5.871349608737935, CurrSamplesPerSec=5.331870586294818, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2360/5000 [8:06:32<8:03:33, 10.99s/it][2022-12-20 18:29:03,011] [INFO] [timer.py:197:stop] 0/4778, RunningAvgSamplesPerSec=5.8713492541808066, CurrSamplesPerSec=5.315323499065961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2361/5000 [8:06:43<8:03:51, 11.00s/it][2022-12-20 18:29:14,035] [INFO] [logging.py:68:log_dist] [Rank 0] step=2390, skipped=3, lr=[5.80888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:29:14,036] [INFO] [timer.py:197:stop] 0/4780, RunningAvgSamplesPerSec=5.871349326091083, CurrSamplesPerSec=5.31751337740555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2362/5000 [8:06:54<8:03:23, 10.99s/it][2022-12-20 18:29:24,962] [INFO] [timer.py:197:stop] 0/4782, RunningAvgSamplesPerSec=5.871371212931361, CurrSamplesPerSec=5.363026885754006, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2363/5000 [8:07:05<8:02:22, 10.98s/it][2022-12-20 18:29:35,898] [INFO] [timer.py:197:stop] 0/4784, RunningAvgSamplesPerSec=5.871390661734643, CurrSamplesPerSec=5.358502886721719, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2364/5000 [8:07:16<8:02:17, 10.98s/it][2022-12-20 18:29:46,887] [INFO] [timer.py:197:stop] 0/4786, RunningAvgSamplesPerSec=5.871398564487362, CurrSamplesPerSec=5.354393186026385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2365/5000 [8:07:27<8:02:12, 10.98s/it][2022-12-20 18:29:57,855] [INFO] [timer.py:197:stop] 0/4788, RunningAvgSamplesPerSec=5.871410604610635, CurrSamplesPerSec=5.3689423468555475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2366/5000 [8:07:38<8:01:37, 10.97s/it][2022-12-20 18:30:08,850] [INFO] [timer.py:197:stop] 0/4790, RunningAvgSamplesPerSec=5.871416844215971, CurrSamplesPerSec=5.328237955783054, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2367/5000 [8:07:49<8:01:59, 10.98s/it][2022-12-20 18:30:19,786] [INFO] [timer.py:197:stop] 0/4792, RunningAvgSamplesPerSec=5.871436365759751, CurrSamplesPerSec=5.397679979801992, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2368/5000 [8:08:00<8:00:39, 10.96s/it][2022-12-20 18:30:30,752] [INFO] [timer.py:197:stop] 0/4794, RunningAvgSamplesPerSec=5.871449434694542, CurrSamplesPerSec=5.336274390450937, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2369/5000 [8:08:11<8:01:32, 10.98s/it][2022-12-20 18:30:41,798] [INFO] [timer.py:197:stop] 0/4796, RunningAvgSamplesPerSec=5.871444179956857, CurrSamplesPerSec=5.328559913051929, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2370/5000 [8:08:22<8:02:07, 11.00s/it][2022-12-20 18:30:52,812] [INFO] [timer.py:197:stop] 0/4798, RunningAvgSamplesPerSec=5.871446201639655, CurrSamplesPerSec=5.352123318007361, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2371/5000 [8:08:33<8:02:13, 11.01s/it][2022-12-20 18:31:03,844] [INFO] [logging.py:68:log_dist] [Rank 0] step=2400, skipped=3, lr=[5.7866666666666674e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:31:03,845] [INFO] [timer.py:197:stop] 0/4800, RunningAvgSamplesPerSec=5.871444561064582, CurrSamplesPerSec=5.339088710090611, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2372/5000 [8:08:44<8:01:32, 10.99s/it][2022-12-20 18:31:14,796] [INFO] [timer.py:197:stop] 0/4802, RunningAvgSamplesPerSec=5.871460546645895, CurrSamplesPerSec=5.35483217950346, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2373/5000 [8:08:55<8:01:19, 10.99s/it][2022-12-20 18:31:25,804] [INFO] [timer.py:197:stop] 0/4804, RunningAvgSamplesPerSec=5.8714638150683625, CurrSamplesPerSec=5.3399487965755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 47%|████▋     | 2374/5000 [8:09:06<8:01:26, 11.00s/it][2022-12-20 18:31:36,838] [INFO] [timer.py:197:stop] 0/4806, RunningAvgSamplesPerSec=5.871461308607143, CurrSamplesPerSec=5.324365672929574, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2375/5000 [8:09:17<8:01:53, 11.01s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.78e-06, 'epoch': 57.92}

 48%|████▊     | 2375/5000 [8:09:17<8:01:53, 11.01s/it][2022-12-20 18:31:47,938] [INFO] [timer.py:197:stop] 0/4808, RunningAvgSamplesPerSec=5.8714518389484756, CurrSamplesPerSec=5.295220775228394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2376/5000 [8:09:28<8:02:18, 11.03s/it][2022-12-20 18:31:58,941] [INFO] [timer.py:197:stop] 0/4810, RunningAvgSamplesPerSec=5.871457473056553, CurrSamplesPerSec=5.330284379178751, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2377/5000 [8:09:39<8:01:56, 11.02s/it][2022-12-20 18:32:09,976] [INFO] [timer.py:197:stop] 0/4812, RunningAvgSamplesPerSec=5.871454481356718, CurrSamplesPerSec=5.310627004872787, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2378/5000 [8:09:50<8:02:04, 11.03s/it][2022-12-20 18:32:20,093] [INFO] [timer.py:197:stop] 0/4814, RunningAvgSamplesPerSec=5.871655140383373, CurrSamplesPerSec=6.271014257244144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 18:32:31,103] [INFO] [timer.py:197:stop] 0/4816, RunningAvgSamplesPerSec=5.871657608221426, CurrSamplesPerSec=5.331562630317045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2379/5000 [8:10:06<9:09:11, 12.57s/it][2022-12-20 18:32:42,115] [INFO] [timer.py:197:stop] 0/4818, RunningAvgSamplesPerSec=5.871660124734972, CurrSamplesPerSec=5.330284590864146, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2380/5000 [8:10:17<8:48:32, 12.10s/it][2022-12-20 18:32:53,122] [INFO] [logging.py:68:log_dist] [Rank 0] step=2410, skipped=3, lr=[5.764444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:32:53,123] [INFO] [timer.py:197:stop] 0/4820, RunningAvgSamplesPerSec=5.871663243961402, CurrSamplesPerSec=5.298652875039607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2381/5000 [8:10:28<8:33:59, 11.78s/it][2022-12-20 18:33:04,101] [INFO] [timer.py:197:stop] 0/4822, RunningAvgSamplesPerSec=5.8716734776601935, CurrSamplesPerSec=5.347792193437963, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2382/5000 [8:10:39<8:23:21, 11.54s/it][2022-12-20 18:33:15,046] [INFO] [timer.py:197:stop] 0/4824, RunningAvgSamplesPerSec=5.871691034025172, CurrSamplesPerSec=5.365108059470483, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2383/5000 [8:10:50<8:15:25, 11.36s/it][2022-12-20 18:33:26,043] [INFO] [timer.py:197:stop] 0/4826, RunningAvgSamplesPerSec=5.871696988808957, CurrSamplesPerSec=5.344771385500332, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2384/5000 [8:11:01<8:10:30, 11.25s/it][2022-12-20 18:33:37,025] [INFO] [timer.py:197:stop] 0/4828, RunningAvgSamplesPerSec=5.871706102369788, CurrSamplesPerSec=5.3414876171459955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2385/5000 [8:11:12<8:06:49, 11.17s/it][2022-12-20 18:33:48,075] [INFO] [timer.py:197:stop] 0/4830, RunningAvgSamplesPerSec=5.871700183288573, CurrSamplesPerSec=5.318114493232603, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2386/5000 [8:11:23<8:05:03, 11.13s/it][2022-12-20 18:33:59,093] [INFO] [timer.py:197:stop] 0/4832, RunningAvgSamplesPerSec=5.871701511435076, CurrSamplesPerSec=5.314612950286145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2387/5000 [8:11:34<8:03:21, 11.10s/it][2022-12-20 18:34:10,082] [INFO] [timer.py:197:stop] 0/4834, RunningAvgSamplesPerSec=5.871710156019766, CurrSamplesPerSec=5.364259573968584, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2388/5000 [8:11:45<8:01:44, 11.07s/it][2022-12-20 18:34:21,101] [INFO] [timer.py:197:stop] 0/4836, RunningAvgSamplesPerSec=5.871711352634429, CurrSamplesPerSec=5.348038949460336, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2389/5000 [8:11:56<8:00:56, 11.05s/it][2022-12-20 18:34:32,084] [INFO] [timer.py:197:stop] 0/4838, RunningAvgSamplesPerSec=5.871720277587745, CurrSamplesPerSec=5.344100179963291, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2390/5000 [8:12:07<7:59:51, 11.03s/it][2022-12-20 18:34:43,109] [INFO] [logging.py:68:log_dist] [Rank 0] step=2420, skipped=3, lr=[5.742222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:34:43,110] [INFO] [timer.py:197:stop] 0/4840, RunningAvgSamplesPerSec=5.87172304075709, CurrSamplesPerSec=5.302100108836711, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2391/5000 [8:12:18<7:59:37, 11.03s/it][2022-12-20 18:34:54,179] [INFO] [timer.py:197:stop] 0/4842, RunningAvgSamplesPerSec=5.871720267588995, CurrSamplesPerSec=5.301456120011009, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2392/5000 [8:12:29<7:59:56, 11.04s/it][2022-12-20 18:35:05,215] [INFO] [timer.py:197:stop] 0/4844, RunningAvgSamplesPerSec=5.87172481922872, CurrSamplesPerSec=5.332558002879916, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2393/5000 [8:12:40<7:59:40, 11.04s/it][2022-12-20 18:35:16,359] [INFO] [timer.py:197:stop] 0/4846, RunningAvgSamplesPerSec=5.871705350974805, CurrSamplesPerSec=5.27148273585709, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2394/5000 [8:12:51<8:00:51, 11.07s/it][2022-12-20 18:35:27,443] [INFO] [timer.py:197:stop] 0/4848, RunningAvgSamplesPerSec=5.871696666494056, CurrSamplesPerSec=5.285085536985909, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2395/5000 [8:13:03<8:00:49, 11.07s/it][2022-12-20 18:35:38,502] [INFO] [timer.py:197:stop] 0/4850, RunningAvgSamplesPerSec=5.871696188018329, CurrSamplesPerSec=5.2976630090122185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2396/5000 [8:13:14<8:00:26, 11.07s/it][2022-12-20 18:35:49,544] [INFO] [timer.py:197:stop] 0/4852, RunningAvgSamplesPerSec=5.871691588914764, CurrSamplesPerSec=5.319879436221168, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2397/5000 [8:13:25<7:59:53, 11.06s/it][2022-12-20 18:36:00,493] [INFO] [timer.py:197:stop] 0/4854, RunningAvgSamplesPerSec=5.871707753738764, CurrSamplesPerSec=5.367595021977238, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2398/5000 [8:13:36<7:58:14, 11.03s/it][2022-12-20 18:36:11,480] [INFO] [timer.py:197:stop] 0/4856, RunningAvgSamplesPerSec=5.871715644285251, CurrSamplesPerSec=5.347717190938689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2399/5000 [8:13:47<7:57:31, 11.02s/it][2022-12-20 18:36:22,496] [INFO] [timer.py:197:stop] 0/4858, RunningAvgSamplesPerSec=5.871717154842123, CurrSamplesPerSec=5.310279897872513, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2400/5000 [8:13:58<7:57:20, 11.02s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.722222222222222e-06, 'epoch': 58.53}

 48%|████▊     | 2400/5000 [8:13:58<7:57:20, 11.02s/it][2022-12-20 18:36:33,595] [INFO] [logging.py:68:log_dist] [Rank 0] step=2430, skipped=3, lr=[5.72e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:36:33,596] [INFO] [timer.py:197:stop] 0/4860, RunningAvgSamplesPerSec=5.8717001262260275, CurrSamplesPerSec=5.255825138376957, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2401/5000 [8:14:09<7:58:15, 11.04s/it][2022-12-20 18:36:44,593] [INFO] [timer.py:197:stop] 0/4862, RunningAvgSamplesPerSec=5.871706038039523, CurrSamplesPerSec=5.3499120968494775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2402/5000 [8:14:20<7:57:30, 11.03s/it][2022-12-20 18:36:55,607] [INFO] [timer.py:197:stop] 0/4864, RunningAvgSamplesPerSec=5.8717080901022936, CurrSamplesPerSec=5.313388667619308, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2403/5000 [8:14:31<7:57:08, 11.02s/it][2022-12-20 18:37:06,552] [INFO] [timer.py:197:stop] 0/4866, RunningAvgSamplesPerSec=5.871725462517289, CurrSamplesPerSec=5.359632261187743, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2404/5000 [8:14:42<7:55:56, 11.00s/it][2022-12-20 18:37:17,596] [INFO] [timer.py:197:stop] 0/4868, RunningAvgSamplesPerSec=5.871721030792308, CurrSamplesPerSec=5.314184523947395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2405/5000 [8:14:53<7:56:19, 11.01s/it][2022-12-20 18:37:28,592] [INFO] [timer.py:197:stop] 0/4870, RunningAvgSamplesPerSec=5.871727067661912, CurrSamplesPerSec=5.3435089182014055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2406/5000 [8:15:04<7:55:55, 11.01s/it][2022-12-20 18:37:39,680] [INFO] [timer.py:197:stop] 0/4872, RunningAvgSamplesPerSec=5.871713148231342, CurrSamplesPerSec=5.30028372912509, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2407/5000 [8:15:15<7:56:45, 11.03s/it][2022-12-20 18:37:50,749] [INFO] [timer.py:197:stop] 0/4874, RunningAvgSamplesPerSec=5.871703509486168, CurrSamplesPerSec=5.297550096185987, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2408/5000 [8:15:26<7:57:03, 11.04s/it][2022-12-20 18:38:01,806] [INFO] [timer.py:197:stop] 0/4876, RunningAvgSamplesPerSec=5.871696029210158, CurrSamplesPerSec=5.305417400363923, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2409/5000 [8:15:37<7:57:03, 11.05s/it][2022-12-20 18:38:12,826] [INFO] [timer.py:197:stop] 0/4878, RunningAvgSamplesPerSec=5.871696849115344, CurrSamplesPerSec=5.296535142367608, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2410/5000 [8:15:48<7:56:30, 11.04s/it][2022-12-20 18:38:23,915] [INFO] [logging.py:68:log_dist] [Rank 0] step=2440, skipped=3, lr=[5.697777777777779e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:38:23,917] [INFO] [timer.py:197:stop] 0/4880, RunningAvgSamplesPerSec=5.87168224840771, CurrSamplesPerSec=5.26478087145876, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2411/5000 [8:15:59<7:57:01, 11.05s/it][2022-12-20 18:38:35,029] [INFO] [timer.py:197:stop] 0/4882, RunningAvgSamplesPerSec=5.871663782166975, CurrSamplesPerSec=5.271885253680091, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2412/5000 [8:16:10<7:57:34, 11.07s/it][2022-12-20 18:38:46,145] [INFO] [timer.py:197:stop] 0/4884, RunningAvgSamplesPerSec=5.871649099797516, CurrSamplesPerSec=5.302700049246964, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2413/5000 [8:16:21<7:57:57, 11.09s/it][2022-12-20 18:38:57,182] [INFO] [timer.py:197:stop] 0/4886, RunningAvgSamplesPerSec=5.87164695081476, CurrSamplesPerSec=5.3149526261486075, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2414/5000 [8:16:32<7:57:08, 11.07s/it][2022-12-20 18:39:08,227] [INFO] [timer.py:197:stop] 0/4888, RunningAvgSamplesPerSec=5.8716424759067785, CurrSamplesPerSec=5.35386221760711, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2415/5000 [8:16:43<7:56:37, 11.06s/it][2022-12-20 18:39:19,513] [INFO] [timer.py:197:stop] 0/4890, RunningAvgSamplesPerSec=5.871584188643315, CurrSamplesPerSec=5.10681120965177, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2416/5000 [8:16:55<7:59:19, 11.13s/it][2022-12-20 18:39:30,513] [INFO] [timer.py:197:stop] 0/4892, RunningAvgSamplesPerSec=5.871589347350429, CurrSamplesPerSec=5.321083717977313, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2417/5000 [8:17:06<7:57:28, 11.09s/it][2022-12-20 18:39:41,580] [INFO] [timer.py:197:stop] 0/4894, RunningAvgSamplesPerSec=5.871580876673432, CurrSamplesPerSec=5.300439878401381, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2418/5000 [8:17:17<7:56:58, 11.08s/it][2022-12-20 18:39:52,550] [INFO] [timer.py:197:stop] 0/4896, RunningAvgSamplesPerSec=5.871592482794839, CurrSamplesPerSec=5.312585269189001, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2419/5000 [8:17:28<7:55:18, 11.05s/it][2022-12-20 18:40:02,676] [INFO] [timer.py:197:stop] 0/4898, RunningAvgSamplesPerSec=5.871787780774248, CurrSamplesPerSec=5.304302999941984, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2420/5000 [8:17:43<8:46:30, 12.24s/it][2022-12-20 18:40:13,613] [INFO] [logging.py:68:log_dist] [Rank 0] step=2450, skipped=3, lr=[5.675555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:40:13,614] [INFO] [timer.py:197:stop] 0/4900, RunningAvgSamplesPerSec=5.871806095287539, CurrSamplesPerSec=5.361284384481619, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2421/5000 [8:17:54<8:29:43, 11.86s/it][2022-12-20 18:40:24,622] [INFO] [timer.py:197:stop] 0/4902, RunningAvgSamplesPerSec=5.871809425351564, CurrSamplesPerSec=5.319108644652957, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2422/5000 [8:18:05<8:19:08, 11.62s/it][2022-12-20 18:40:35,636] [INFO] [timer.py:197:stop] 0/4904, RunningAvgSamplesPerSec=5.8718118034632125, CurrSamplesPerSec=5.353556413926023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2423/5000 [8:18:16<8:10:34, 11.42s/it][2022-12-20 18:40:46,613] [INFO] [timer.py:197:stop] 0/4906, RunningAvgSamplesPerSec=5.87182160179414, CurrSamplesPerSec=5.343865914871038, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2424/5000 [8:18:27<8:05:00, 11.30s/it][2022-12-20 18:40:57,662] [INFO] [timer.py:197:stop] 0/4908, RunningAvgSamplesPerSec=5.8718161706456735, CurrSamplesPerSec=5.305903983492042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 48%|████▊     | 2425/5000 [8:18:38<8:01:16, 11.21s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.666666666666667e-06, 'epoch': 59.14}

 48%|████▊     | 2425/5000 [8:18:38<8:01:16, 11.21s/it][2022-12-20 18:41:08,722] [INFO] [timer.py:197:stop] 0/4910, RunningAvgSamplesPerSec=5.871808174117076, CurrSamplesPerSec=5.272714708738472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2426/5000 [8:18:49<7:58:58, 11.16s/it][2022-12-20 18:41:19,721] [INFO] [timer.py:197:stop] 0/4912, RunningAvgSamplesPerSec=5.871813841561986, CurrSamplesPerSec=5.316887543864966, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2427/5000 [8:19:00<7:57:34, 11.14s/it][2022-12-20 18:41:30,908] [INFO] [timer.py:197:stop] 0/4914, RunningAvgSamplesPerSec=5.871783117285293, CurrSamplesPerSec=5.233740172338104, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2428/5000 [8:19:11<7:57:34, 11.14s/it][2022-12-20 18:41:41,919] [INFO] [timer.py:197:stop] 0/4916, RunningAvgSamplesPerSec=5.871785734102461, CurrSamplesPerSec=5.336652064514544, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2429/5000 [8:19:22<7:55:42, 11.10s/it][2022-12-20 18:41:52,968] [INFO] [timer.py:197:stop] 0/4918, RunningAvgSamplesPerSec=5.871780106738351, CurrSamplesPerSec=5.303191162968575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2430/5000 [8:19:33<7:54:51, 11.09s/it][2022-12-20 18:42:04,016] [INFO] [logging.py:68:log_dist] [Rank 0] step=2460, skipped=3, lr=[5.653333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:42:04,017] [INFO] [timer.py:197:stop] 0/4920, RunningAvgSamplesPerSec=5.871774320787825, CurrSamplesPerSec=5.303234956962544, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2431/5000 [8:19:44<7:54:21, 11.08s/it][2022-12-20 18:42:15,055] [INFO] [timer.py:197:stop] 0/4922, RunningAvgSamplesPerSec=5.871770766148264, CurrSamplesPerSec=5.323737380849016, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2432/5000 [8:19:55<7:53:24, 11.06s/it][2022-12-20 18:42:26,149] [INFO] [timer.py:197:stop] 0/4924, RunningAvgSamplesPerSec=5.871755244367677, CurrSamplesPerSec=5.258758162928191, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2433/5000 [8:20:06<7:53:47, 11.07s/it][2022-12-20 18:42:37,396] [INFO] [timer.py:197:stop] 0/4926, RunningAvgSamplesPerSec=5.871707256697994, CurrSamplesPerSec=5.143848065126159, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2434/5000 [8:20:18<7:58:07, 11.18s/it][2022-12-20 18:42:48,567] [INFO] [timer.py:197:stop] 0/4928, RunningAvgSamplesPerSec=5.871713586246785, CurrSamplesPerSec=5.35856364428101, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2435/5000 [8:20:29<7:55:27, 11.12s/it][2022-12-20 18:42:59,715] [INFO] [timer.py:197:stop] 0/4930, RunningAvgSamplesPerSec=5.871696605304927, CurrSamplesPerSec=5.259028298337791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2436/5000 [8:20:40<7:55:09, 11.12s/it][2022-12-20 18:43:10,720] [INFO] [timer.py:197:stop] 0/4932, RunningAvgSamplesPerSec=5.871700534574397, CurrSamplesPerSec=5.311321355023245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▊     | 2437/5000 [8:20:51<7:54:12, 11.10s/it][2022-12-20 18:43:21,764] [INFO] [timer.py:197:stop] 0/4934, RunningAvgSamplesPerSec=5.8716965673483665, CurrSamplesPerSec=5.326587742513263, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2438/5000 [8:21:02<7:52:31, 11.07s/it][2022-12-20 18:43:32,797] [INFO] [timer.py:197:stop] 0/4936, RunningAvgSamplesPerSec=5.8716945004175445, CurrSamplesPerSec=5.281912508136298, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2439/5000 [8:21:13<7:52:24, 11.07s/it][2022-12-20 18:43:43,851] [INFO] [timer.py:197:stop] 0/4938, RunningAvgSamplesPerSec=5.871688158893296, CurrSamplesPerSec=5.298625263366504, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2440/5000 [8:21:24<7:52:03, 11.06s/it][2022-12-20 18:43:54,923] [INFO] [logging.py:68:log_dist] [Rank 0] step=2470, skipped=3, lr=[5.631111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:43:54,925] [INFO] [timer.py:197:stop] 0/4940, RunningAvgSamplesPerSec=5.871679879289307, CurrSamplesPerSec=5.290846066412871, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2441/5000 [8:21:35<7:52:30, 11.08s/it][2022-12-20 18:44:05,936] [INFO] [timer.py:197:stop] 0/4942, RunningAvgSamplesPerSec=5.8716827241793474, CurrSamplesPerSec=5.372342112630245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2442/5000 [8:21:46<7:51:09, 11.05s/it][2022-12-20 18:44:16,953] [INFO] [timer.py:197:stop] 0/4944, RunningAvgSamplesPerSec=5.871684243741934, CurrSamplesPerSec=5.345195178792876, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2443/5000 [8:21:57<7:50:36, 11.04s/it][2022-12-20 18:44:28,023] [INFO] [timer.py:197:stop] 0/4946, RunningAvgSamplesPerSec=5.8716756842036855, CurrSamplesPerSec=5.303326319010396, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2444/5000 [8:22:08<7:50:44, 11.05s/it][2022-12-20 18:44:39,052] [INFO] [timer.py:197:stop] 0/4948, RunningAvgSamplesPerSec=5.871674754827284, CurrSamplesPerSec=5.337118926558458, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2445/5000 [8:22:19<7:49:46, 11.03s/it][2022-12-20 18:44:50,095] [INFO] [timer.py:197:stop] 0/4950, RunningAvgSamplesPerSec=5.8716705464871355, CurrSamplesPerSec=5.289995468625582, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2446/5000 [8:22:30<7:49:48, 11.04s/it][2022-12-20 18:45:01,183] [INFO] [timer.py:197:stop] 0/4952, RunningAvgSamplesPerSec=5.871658834274283, CurrSamplesPerSec=5.257137521243159, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2447/5000 [8:22:41<7:50:11, 11.05s/it][2022-12-20 18:45:12,438] [INFO] [timer.py:197:stop] 0/4954, RunningAvgSamplesPerSec=5.87160896367958, CurrSamplesPerSec=5.110452018413321, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2448/5000 [8:22:52<7:52:35, 11.11s/it][2022-12-20 18:45:23,493] [INFO] [timer.py:197:stop] 0/4956, RunningAvgSamplesPerSec=5.871602054551806, CurrSamplesPerSec=5.276547464550864, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2449/5000 [8:23:04<7:51:31, 11.09s/it][2022-12-20 18:45:34,478] [INFO] [timer.py:197:stop] 0/4958, RunningAvgSamplesPerSec=5.871610235201246, CurrSamplesPerSec=5.326988571665799, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2450/5000 [8:23:14<7:49:53, 11.06s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.611111111111112e-06, 'epoch': 59.75}

 49%|████▉     | 2450/5000 [8:23:14<7:49:53, 11.06s/it][2022-12-20 18:45:45,463] [INFO] [logging.py:68:log_dist] [Rank 0] step=2480, skipped=3, lr=[5.608888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:45:45,465] [INFO] [timer.py:197:stop] 0/4960, RunningAvgSamplesPerSec=5.871618295347072, CurrSamplesPerSec=5.3191834792289345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2451/5000 [8:23:26<7:49:27, 11.05s/it][2022-12-20 18:45:56,513] [INFO] [timer.py:197:stop] 0/4962, RunningAvgSamplesPerSec=5.87161284404724, CurrSamplesPerSec=5.308229068950839, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2452/5000 [8:23:37<7:49:13, 11.05s/it][2022-12-20 18:46:07,542] [INFO] [timer.py:197:stop] 0/4964, RunningAvgSamplesPerSec=5.87161236718972, CurrSamplesPerSec=5.326316751918808, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2453/5000 [8:23:48<7:48:24, 11.03s/it][2022-12-20 18:46:18,548] [INFO] [timer.py:197:stop] 0/4966, RunningAvgSamplesPerSec=5.871616230619212, CurrSamplesPerSec=5.319873321295308, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2454/5000 [8:23:59<7:48:04, 11.03s/it][2022-12-20 18:46:29,588] [INFO] [timer.py:197:stop] 0/4968, RunningAvgSamplesPerSec=5.871613727343084, CurrSamplesPerSec=5.304967809778389, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2455/5000 [8:24:10<7:48:11, 11.04s/it][2022-12-20 18:46:40,579] [INFO] [timer.py:197:stop] 0/4970, RunningAvgSamplesPerSec=5.871620842556244, CurrSamplesPerSec=5.359499356306124, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2456/5000 [8:24:21<7:47:27, 11.03s/it][2022-12-20 18:46:51,638] [INFO] [timer.py:197:stop] 0/4972, RunningAvgSamplesPerSec=5.8716141009507945, CurrSamplesPerSec=5.305595034808724, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2457/5000 [8:24:32<7:47:14, 11.02s/it][2022-12-20 18:47:02,608] [INFO] [timer.py:197:stop] 0/4974, RunningAvgSamplesPerSec=5.871626238084978, CurrSamplesPerSec=5.349131299444035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2458/5000 [8:24:43<7:46:08, 11.00s/it][2022-12-20 18:47:13,577] [INFO] [timer.py:197:stop] 0/4976, RunningAvgSamplesPerSec=5.871638273895937, CurrSamplesPerSec=5.333032413084245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2459/5000 [8:24:54<7:45:48, 11.00s/it][2022-12-20 18:47:24,581] [INFO] [timer.py:197:stop] 0/4978, RunningAvgSamplesPerSec=5.871642987706822, CurrSamplesPerSec=5.3226904161080295, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2460/5000 [8:25:05<7:46:01, 11.01s/it][2022-12-20 18:47:34,733] [INFO] [logging.py:68:log_dist] [Rank 0] step=2490, skipped=3, lr=[5.586666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:47:34,734] [INFO] [timer.py:197:stop] 0/4980, RunningAvgSamplesPerSec=5.871829592580814, CurrSamplesPerSec=6.221628139850031, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 18:47:45,724] [INFO] [timer.py:197:stop] 0/4982, RunningAvgSamplesPerSec=5.871836648371863, CurrSamplesPerSec=5.3329696903233295, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2461/5000 [8:25:21<8:51:36, 12.56s/it][2022-12-20 18:47:56,738] [INFO] [timer.py:197:stop] 0/4984, RunningAvgSamplesPerSec=5.871839216278782, CurrSamplesPerSec=5.3476647756682, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2462/5000 [8:25:32<8:31:45, 12.10s/it][2022-12-20 18:48:07,770] [INFO] [timer.py:197:stop] 0/4986, RunningAvgSamplesPerSec=5.871837500025651, CurrSamplesPerSec=5.336332735480601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2463/5000 [8:25:43<8:18:01, 11.78s/it][2022-12-20 18:48:18,789] [INFO] [timer.py:197:stop] 0/4988, RunningAvgSamplesPerSec=5.871838286040147, CurrSamplesPerSec=5.331701354363256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2464/5000 [8:25:54<8:08:12, 11.55s/it][2022-12-20 18:48:29,798] [INFO] [timer.py:197:stop] 0/4990, RunningAvgSamplesPerSec=5.8718422770787075, CurrSamplesPerSec=5.316900602501156, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2465/5000 [8:26:05<8:01:08, 11.39s/it][2022-12-20 18:48:40,783] [INFO] [timer.py:197:stop] 0/4992, RunningAvgSamplesPerSec=5.871850459880945, CurrSamplesPerSec=5.3382824050105135, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2466/5000 [8:26:16<7:55:50, 11.27s/it][2022-12-20 18:48:51,806] [INFO] [timer.py:197:stop] 0/4994, RunningAvgSamplesPerSec=5.871850797416019, CurrSamplesPerSec=5.301406911039251, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2467/5000 [8:26:27<7:52:34, 11.19s/it][2022-12-20 18:49:02,823] [INFO] [timer.py:197:stop] 0/4996, RunningAvgSamplesPerSec=5.871853261782841, CurrSamplesPerSec=5.310736693395954, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2468/5000 [8:26:38<7:50:08, 11.14s/it][2022-12-20 18:49:13,901] [INFO] [timer.py:197:stop] 0/4998, RunningAvgSamplesPerSec=5.871847210848844, CurrSamplesPerSec=5.302028476980958, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2469/5000 [8:26:49<7:49:09, 11.12s/it][2022-12-20 18:49:24,954] [INFO] [logging.py:68:log_dist] [Rank 0] step=2500, skipped=3, lr=[5.5644444444444444e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:49:24,955] [INFO] [timer.py:197:stop] 0/5000, RunningAvgSamplesPerSec=5.871841508671049, CurrSamplesPerSec=5.315770004418398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2470/5000 [8:27:00<7:48:07, 11.10s/it][2022-12-20 18:49:36,007] [INFO] [timer.py:197:stop] 0/5002, RunningAvgSamplesPerSec=5.871835410737261, CurrSamplesPerSec=5.329846014798496, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2471/5000 [8:27:11<7:47:17, 11.09s/it][2022-12-20 18:49:47,045] [INFO] [timer.py:197:stop] 0/5004, RunningAvgSamplesPerSec=5.871831986780581, CurrSamplesPerSec=5.310777460017343, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2472/5000 [8:27:22<7:46:30, 11.07s/it][2022-12-20 18:49:58,028] [INFO] [timer.py:197:stop] 0/5006, RunningAvgSamplesPerSec=5.871840397442546, CurrSamplesPerSec=5.286396126111237, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2473/5000 [8:27:33<7:45:12, 11.05s/it][2022-12-20 18:50:09,070] [INFO] [timer.py:197:stop] 0/5008, RunningAvgSamplesPerSec=5.871840993962608, CurrSamplesPerSec=5.303263035788978, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 49%|████▉     | 2474/5000 [8:27:44<7:44:58, 11.04s/it][2022-12-20 18:50:20,093] [INFO] [timer.py:197:stop] 0/5010, RunningAvgSamplesPerSec=5.8718468332763765, CurrSamplesPerSec=5.333661205525595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2475/5000 [8:27:55<7:44:30, 11.04s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.553333333333334e-06, 'epoch': 60.36}

 50%|████▉     | 2475/5000 [8:27:55<7:44:30, 11.04s/it][2022-12-20 18:50:31,141] [INFO] [timer.py:197:stop] 0/5012, RunningAvgSamplesPerSec=5.871841746152169, CurrSamplesPerSec=5.3116582979523495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2476/5000 [8:28:06<7:44:27, 11.04s/it][2022-12-20 18:50:42,142] [INFO] [timer.py:197:stop] 0/5014, RunningAvgSamplesPerSec=5.8718466335876816, CurrSamplesPerSec=5.324295550294023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2477/5000 [8:28:17<7:43:45, 11.03s/it][2022-12-20 18:50:53,167] [INFO] [timer.py:197:stop] 0/5016, RunningAvgSamplesPerSec=5.871846155118186, CurrSamplesPerSec=5.335796857821585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2478/5000 [8:28:28<7:43:31, 11.03s/it][2022-12-20 18:51:04,162] [INFO] [timer.py:197:stop] 0/5018, RunningAvgSamplesPerSec=5.871852185824686, CurrSamplesPerSec=5.360985655186286, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2479/5000 [8:28:39<7:42:55, 11.02s/it][2022-12-20 18:51:15,198] [INFO] [logging.py:68:log_dist] [Rank 0] step=2510, skipped=3, lr=[5.542222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:51:15,199] [INFO] [timer.py:197:stop] 0/5020, RunningAvgSamplesPerSec=5.87184903883532, CurrSamplesPerSec=5.3179115777435415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2480/5000 [8:28:50<7:42:59, 11.02s/it][2022-12-20 18:51:26,190] [INFO] [timer.py:197:stop] 0/5022, RunningAvgSamplesPerSec=5.8718560005630245, CurrSamplesPerSec=5.322349961576678, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2481/5000 [8:29:01<7:42:23, 11.01s/it][2022-12-20 18:51:37,126] [INFO] [timer.py:197:stop] 0/5024, RunningAvgSamplesPerSec=5.871874402778357, CurrSamplesPerSec=5.3861767065062685, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2482/5000 [8:29:12<7:41:14, 10.99s/it][2022-12-20 18:51:48,160] [INFO] [timer.py:197:stop] 0/5026, RunningAvgSamplesPerSec=5.8718721569956145, CurrSamplesPerSec=5.312690201942249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2483/5000 [8:29:23<7:41:35, 11.00s/it][2022-12-20 18:51:59,114] [INFO] [timer.py:197:stop] 0/5028, RunningAvgSamplesPerSec=5.871886978181, CurrSamplesPerSec=5.356709020452382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2484/5000 [8:29:34<7:40:47, 10.99s/it][2022-12-20 18:52:10,138] [INFO] [timer.py:197:stop] 0/5030, RunningAvgSamplesPerSec=5.871886662021788, CurrSamplesPerSec=5.323994804277007, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2485/5000 [8:29:45<7:41:03, 11.00s/it][2022-12-20 18:52:21,139] [INFO] [timer.py:197:stop] 0/5032, RunningAvgSamplesPerSec=5.87189140402149, CurrSamplesPerSec=5.30809744158186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2486/5000 [8:29:56<7:40:53, 11.00s/it][2022-12-20 18:52:32,171] [INFO] [timer.py:197:stop] 0/5034, RunningAvgSamplesPerSec=5.8718893024745045, CurrSamplesPerSec=5.302125452734595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2487/5000 [8:30:07<7:41:06, 11.01s/it][2022-12-20 18:52:43,143] [INFO] [timer.py:197:stop] 0/5036, RunningAvgSamplesPerSec=5.871900254620668, CurrSamplesPerSec=5.34094475279706, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2488/5000 [8:30:18<7:40:27, 11.00s/it][2022-12-20 18:52:54,171] [INFO] [timer.py:197:stop] 0/5038, RunningAvgSamplesPerSec=5.871898983053507, CurrSamplesPerSec=5.317138828558471, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2489/5000 [8:30:29<7:40:39, 11.01s/it][2022-12-20 18:53:05,118] [INFO] [logging.py:68:log_dist] [Rank 0] step=2520, skipped=3, lr=[5.5200000000000005e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:53:05,119] [INFO] [timer.py:197:stop] 0/5040, RunningAvgSamplesPerSec=5.871914660040124, CurrSamplesPerSec=5.353910910244085, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2490/5000 [8:30:40<7:39:43, 10.99s/it][2022-12-20 18:53:16,171] [INFO] [timer.py:197:stop] 0/5042, RunningAvgSamplesPerSec=5.871908439166753, CurrSamplesPerSec=5.330472150730607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2491/5000 [8:30:51<7:40:19, 11.01s/it][2022-12-20 18:53:27,136] [INFO] [timer.py:197:stop] 0/5044, RunningAvgSamplesPerSec=5.871920965391994, CurrSamplesPerSec=5.356706454980454, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2492/5000 [8:31:02<7:39:35, 11.00s/it][2022-12-20 18:53:38,098] [INFO] [timer.py:197:stop] 0/5046, RunningAvgSamplesPerSec=5.871933840949352, CurrSamplesPerSec=5.341326064146504, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2493/5000 [8:31:13<7:38:59, 10.99s/it][2022-12-20 18:53:49,146] [INFO] [timer.py:197:stop] 0/5048, RunningAvgSamplesPerSec=5.8719282825272625, CurrSamplesPerSec=5.301406073447639, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2494/5000 [8:31:24<7:39:36, 11.00s/it][2022-12-20 18:54:00,154] [INFO] [timer.py:197:stop] 0/5050, RunningAvgSamplesPerSec=5.871931482356053, CurrSamplesPerSec=5.323001359010395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2495/5000 [8:31:35<7:39:27, 11.01s/it][2022-12-20 18:54:11,197] [INFO] [timer.py:197:stop] 0/5052, RunningAvgSamplesPerSec=5.871927496777105, CurrSamplesPerSec=5.307984923200993, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2496/5000 [8:31:46<7:39:45, 11.02s/it][2022-12-20 18:54:22,196] [INFO] [timer.py:197:stop] 0/5054, RunningAvgSamplesPerSec=5.871932283640357, CurrSamplesPerSec=5.309838095461523, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2497/5000 [8:31:57<7:39:21, 11.01s/it][2022-12-20 18:54:33,218] [INFO] [timer.py:197:stop] 0/5056, RunningAvgSamplesPerSec=5.871932475939159, CurrSamplesPerSec=5.3038085364627205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2498/5000 [8:32:08<7:39:18, 11.01s/it][2022-12-20 18:54:44,177] [INFO] [timer.py:197:stop] 0/5058, RunningAvgSamplesPerSec=5.8719465345984885, CurrSamplesPerSec=5.3673674928214945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|████▉     | 2499/5000 [8:32:19<7:38:25, 11.00s/it][2022-12-20 18:54:55,182] [INFO] [logging.py:68:log_dist] [Rank 0] step=2530, skipped=3, lr=[5.497777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:54:55,184] [INFO] [timer.py:197:stop] 0/5060, RunningAvgSamplesPerSec=5.871950117571574, CurrSamplesPerSec=5.336104454561736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2500/5000 [8:32:30<7:38:21, 11.00s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.497777777777778e-06, 'epoch': 60.96}

 50%|█████     | 2500/5000 [8:32:30<7:38:21, 11.00s/it][2022-12-20 18:55:06,176] [INFO] [timer.py:197:stop] 0/5062, RunningAvgSamplesPerSec=5.871957127467681, CurrSamplesPerSec=5.349127675301577, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2501/5000 [8:32:41<7:38:04, 11.00s/it][2022-12-20 18:55:16,326] [INFO] [timer.py:197:stop] 0/5064, RunningAvgSamplesPerSec=5.8721411534671635, CurrSamplesPerSec=5.306683961061043, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2502/5000 [8:32:56<8:28:49, 12.22s/it][2022-12-20 18:55:27,354] [INFO] [timer.py:197:stop] 0/5066, RunningAvgSamplesPerSec=5.872139924419976, CurrSamplesPerSec=5.300541191968221, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2503/5000 [8:33:07<8:13:31, 11.86s/it][2022-12-20 18:55:38,344] [INFO] [timer.py:197:stop] 0/5068, RunningAvgSamplesPerSec=5.872146819936066, CurrSamplesPerSec=5.320431946155144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2504/5000 [8:33:18<8:02:57, 11.61s/it][2022-12-20 18:55:49,392] [INFO] [timer.py:197:stop] 0/5070, RunningAvgSamplesPerSec=5.872141464133654, CurrSamplesPerSec=5.302871844896868, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2505/5000 [8:33:29<7:55:10, 11.43s/it][2022-12-20 18:56:00,381] [INFO] [timer.py:197:stop] 0/5072, RunningAvgSamplesPerSec=5.872148456066426, CurrSamplesPerSec=5.31271312367645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2506/5000 [8:33:40<7:49:40, 11.30s/it][2022-12-20 18:56:11,343] [INFO] [timer.py:197:stop] 0/5074, RunningAvgSamplesPerSec=5.872161557349289, CurrSamplesPerSec=5.349148354298003, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2507/5000 [8:33:51<7:45:10, 11.20s/it][2022-12-20 18:56:22,345] [INFO] [timer.py:197:stop] 0/5076, RunningAvgSamplesPerSec=5.87216604640883, CurrSamplesPerSec=5.305302269397074, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2508/5000 [8:34:02<7:43:09, 11.15s/it][2022-12-20 18:56:33,414] [INFO] [timer.py:197:stop] 0/5078, RunningAvgSamplesPerSec=5.872156162464104, CurrSamplesPerSec=5.28726243748894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2509/5000 [8:34:13<7:41:45, 11.12s/it][2022-12-20 18:56:44,412] [INFO] [logging.py:68:log_dist] [Rank 0] step=2540, skipped=3, lr=[5.475555555555557e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:56:44,414] [INFO] [timer.py:197:stop] 0/5080, RunningAvgSamplesPerSec=5.87216086785783, CurrSamplesPerSec=5.3347377235658415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2510/5000 [8:34:24<7:40:09, 11.09s/it][2022-12-20 18:56:55,464] [INFO] [timer.py:197:stop] 0/5082, RunningAvgSamplesPerSec=5.872155032888189, CurrSamplesPerSec=5.298654339303033, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2511/5000 [8:34:36<7:39:38, 11.08s/it][2022-12-20 18:57:06,502] [INFO] [timer.py:197:stop] 0/5084, RunningAvgSamplesPerSec=5.87215175607283, CurrSamplesPerSec=5.3195743396543484, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2512/5000 [8:34:47<7:39:04, 11.07s/it][2022-12-20 18:57:17,540] [INFO] [timer.py:197:stop] 0/5086, RunningAvgSamplesPerSec=5.872148105415894, CurrSamplesPerSec=5.328507238011519, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2513/5000 [8:34:58<7:38:20, 11.06s/it][2022-12-20 18:57:28,576] [INFO] [timer.py:197:stop] 0/5088, RunningAvgSamplesPerSec=5.872145086931538, CurrSamplesPerSec=5.3201925813886906, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2514/5000 [8:35:09<7:37:53, 11.05s/it][2022-12-20 18:57:39,656] [INFO] [timer.py:197:stop] 0/5090, RunningAvgSamplesPerSec=5.8721323002822405, CurrSamplesPerSec=5.281010338671472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2515/5000 [8:35:20<7:37:36, 11.05s/it][2022-12-20 18:57:50,649] [INFO] [timer.py:197:stop] 0/5092, RunningAvgSamplesPerSec=5.872138302153605, CurrSamplesPerSec=5.326531090093715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2516/5000 [8:35:31<7:36:30, 11.03s/it][2022-12-20 18:58:01,651] [INFO] [timer.py:197:stop] 0/5094, RunningAvgSamplesPerSec=5.87214255921676, CurrSamplesPerSec=5.303132702256971, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2517/5000 [8:35:42<7:35:53, 11.02s/it][2022-12-20 18:58:12,630] [INFO] [timer.py:197:stop] 0/5096, RunningAvgSamplesPerSec=5.8721514506222, CurrSamplesPerSec=5.314450283232289, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2518/5000 [8:35:53<7:35:37, 11.01s/it][2022-12-20 18:58:23,633] [INFO] [timer.py:197:stop] 0/5098, RunningAvgSamplesPerSec=5.872155819642922, CurrSamplesPerSec=5.321056082985459, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2519/5000 [8:36:04<7:35:40, 11.02s/it][2022-12-20 18:58:34,659] [INFO] [logging.py:68:log_dist] [Rank 0] step=2550, skipped=3, lr=[5.453333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 18:58:34,660] [INFO] [timer.py:197:stop] 0/5100, RunningAvgSamplesPerSec=5.87215482207691, CurrSamplesPerSec=5.325843958384067, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2520/5000 [8:36:15<7:35:11, 11.01s/it][2022-12-20 18:58:45,627] [INFO] [timer.py:197:stop] 0/5102, RunningAvgSamplesPerSec=5.87216673274909, CurrSamplesPerSec=5.3520074315288735, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2521/5000 [8:36:26<7:34:17, 11.00s/it][2022-12-20 18:58:56,612] [INFO] [timer.py:197:stop] 0/5104, RunningAvgSamplesPerSec=5.872174699293519, CurrSamplesPerSec=5.325479645487349, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2522/5000 [8:36:37<7:33:51, 10.99s/it][2022-12-20 18:59:07,568] [INFO] [timer.py:197:stop] 0/5106, RunningAvgSamplesPerSec=5.8721887528437, CurrSamplesPerSec=5.342204094915323, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2523/5000 [8:36:48<7:33:29, 10.98s/it][2022-12-20 18:59:18,548] [INFO] [timer.py:197:stop] 0/5108, RunningAvgSamplesPerSec=5.872197665123957, CurrSamplesPerSec=5.337070963329383, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2524/5000 [8:36:59<7:33:19, 10.99s/it][2022-12-20 18:59:29,558] [INFO] [timer.py:197:stop] 0/5110, RunningAvgSamplesPerSec=5.872200169816816, CurrSamplesPerSec=5.316130463266113, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 50%|█████     | 2525/5000 [8:37:10<7:33:39, 11.00s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.442222222222223e-06, 'epoch': 61.58}

 50%|█████     | 2525/5000 [8:37:10<7:33:39, 11.00s/it][2022-12-20 18:59:40,556] [INFO] [timer.py:197:stop] 0/5112, RunningAvgSamplesPerSec=5.872205569878199, CurrSamplesPerSec=5.343376386287666, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2526/5000 [8:37:21<7:33:34, 11.00s/it][2022-12-20 18:59:51,582] [INFO] [timer.py:197:stop] 0/5114, RunningAvgSamplesPerSec=5.872204758302169, CurrSamplesPerSec=5.32411518308491, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2527/5000 [8:37:32<7:33:10, 10.99s/it][2022-12-20 19:00:02,515] [INFO] [timer.py:197:stop] 0/5116, RunningAvgSamplesPerSec=5.872223380910028, CurrSamplesPerSec=5.367832445604984, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2528/5000 [8:37:43<7:32:14, 10.98s/it][2022-12-20 19:00:13,487] [INFO] [timer.py:197:stop] 0/5118, RunningAvgSamplesPerSec=5.872234028180447, CurrSamplesPerSec=5.334728393855094, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2529/5000 [8:37:54<7:32:08, 10.98s/it][2022-12-20 19:00:24,492] [INFO] [logging.py:68:log_dist] [Rank 0] step=2560, skipped=3, lr=[5.431111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:00:24,494] [INFO] [timer.py:197:stop] 0/5120, RunningAvgSamplesPerSec=5.872238550734601, CurrSamplesPerSec=5.314313507517374, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2530/5000 [8:38:05<7:32:44, 11.00s/it][2022-12-20 19:00:35,566] [INFO] [timer.py:197:stop] 0/5122, RunningAvgSamplesPerSec=5.872228351084889, CurrSamplesPerSec=5.288056736269761, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2531/5000 [8:38:16<7:33:22, 11.02s/it][2022-12-20 19:00:46,631] [INFO] [timer.py:197:stop] 0/5124, RunningAvgSamplesPerSec=5.8722193601088515, CurrSamplesPerSec=5.286901301075573, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2532/5000 [8:38:27<7:34:00, 11.04s/it][2022-12-20 19:00:57,692] [INFO] [timer.py:197:stop] 0/5126, RunningAvgSamplesPerSec=5.872211210508516, CurrSamplesPerSec=5.306885810783267, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2533/5000 [8:38:38<7:34:09, 11.05s/it][2022-12-20 19:01:08,719] [INFO] [timer.py:197:stop] 0/5128, RunningAvgSamplesPerSec=5.872210355592526, CurrSamplesPerSec=5.340641485485858, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2534/5000 [8:38:49<7:33:32, 11.04s/it][2022-12-20 19:01:19,795] [INFO] [timer.py:197:stop] 0/5130, RunningAvgSamplesPerSec=5.872198833737548, CurrSamplesPerSec=5.282010828108723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2535/5000 [8:39:00<7:33:35, 11.04s/it][2022-12-20 19:01:30,821] [INFO] [timer.py:197:stop] 0/5132, RunningAvgSamplesPerSec=5.872198801741341, CurrSamplesPerSec=5.310039554965101, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2536/5000 [8:39:11<7:33:12, 11.04s/it][2022-12-20 19:01:41,820] [INFO] [timer.py:197:stop] 0/5134, RunningAvgSamplesPerSec=5.872203713309893, CurrSamplesPerSec=5.329619135046385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2537/5000 [8:39:22<7:33:00, 11.04s/it][2022-12-20 19:01:52,889] [INFO] [timer.py:197:stop] 0/5136, RunningAvgSamplesPerSec=5.87219410798, CurrSamplesPerSec=5.299740208828777, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2538/5000 [8:39:33<7:33:04, 11.04s/it][2022-12-20 19:02:04,005] [INFO] [timer.py:197:stop] 0/5138, RunningAvgSamplesPerSec=5.872174337225317, CurrSamplesPerSec=5.2465735057703515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2539/5000 [8:39:44<7:33:49, 11.06s/it][2022-12-20 19:02:15,037] [INFO] [logging.py:68:log_dist] [Rank 0] step=2570, skipped=3, lr=[5.4088888888888894e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:02:15,038] [INFO] [timer.py:197:stop] 0/5140, RunningAvgSamplesPerSec=5.872172191848566, CurrSamplesPerSec=5.320590128524832, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2540/5000 [8:39:55<7:32:55, 11.05s/it][2022-12-20 19:02:26,013] [INFO] [timer.py:197:stop] 0/5142, RunningAvgSamplesPerSec=5.872182161851613, CurrSamplesPerSec=5.348704323983782, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2541/5000 [8:40:06<7:32:19, 11.04s/it][2022-12-20 19:02:37,052] [INFO] [timer.py:197:stop] 0/5144, RunningAvgSamplesPerSec=5.872179133580794, CurrSamplesPerSec=5.3260254994830625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2542/5000 [8:40:17<7:31:41, 11.03s/it][2022-12-20 19:02:47,132] [INFO] [timer.py:197:stop] 0/5146, RunningAvgSamplesPerSec=5.8723747005393445, CurrSamplesPerSec=6.277143622139676, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 19:02:58,124] [INFO] [timer.py:197:stop] 0/5148, RunningAvgSamplesPerSec=5.872380494293966, CurrSamplesPerSec=5.333861297309065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2543/5000 [8:40:33<8:34:23, 12.56s/it][2022-12-20 19:03:09,173] [INFO] [timer.py:197:stop] 0/5150, RunningAvgSamplesPerSec=5.872375238368492, CurrSamplesPerSec=5.320558280384848, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2544/5000 [8:40:44<8:15:36, 12.11s/it][2022-12-20 19:03:20,092] [INFO] [timer.py:197:stop] 0/5152, RunningAvgSamplesPerSec=5.872396955559943, CurrSamplesPerSec=5.382331154271723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2545/5000 [8:40:55<8:00:48, 11.75s/it][2022-12-20 19:03:31,119] [INFO] [timer.py:197:stop] 0/5154, RunningAvgSamplesPerSec=5.872396192332469, CurrSamplesPerSec=5.3003130325775345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2546/5000 [8:41:06<7:51:44, 11.53s/it][2022-12-20 19:03:42,106] [INFO] [timer.py:197:stop] 0/5156, RunningAvgSamplesPerSec=5.872406038616784, CurrSamplesPerSec=5.374869580518407, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2547/5000 [8:41:17<7:44:50, 11.37s/it][2022-12-20 19:03:53,135] [INFO] [timer.py:197:stop] 0/5158, RunningAvgSamplesPerSec=5.872404705316606, CurrSamplesPerSec=5.298845118831041, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2548/5000 [8:41:28<7:40:27, 11.27s/it][2022-12-20 19:04:04,140] [INFO] [logging.py:68:log_dist] [Rank 0] step=2580, skipped=3, lr=[5.386666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:04:04,140] [INFO] [timer.py:197:stop] 0/5160, RunningAvgSamplesPerSec=5.87240817226454, CurrSamplesPerSec=5.335408063861249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2549/5000 [8:41:39<7:37:04, 11.19s/it][2022-12-20 19:04:15,150] [INFO] [timer.py:197:stop] 0/5162, RunningAvgSamplesPerSec=5.872412520415067, CurrSamplesPerSec=5.345415296852599, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2550/5000 [8:41:50<7:34:40, 11.13s/it]
                                                       
{'loss': 0.0002, 'learning_rate': 5.3844444444444445e-06, 'epoch': 62.19}

 51%|█████     | 2550/5000 [8:41:50<7:34:40, 11.13s/it][2022-12-20 19:04:26,209] [INFO] [timer.py:197:stop] 0/5164, RunningAvgSamplesPerSec=5.872405017978086, CurrSamplesPerSec=5.287594459931749, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2551/5000 [8:42:01<7:33:33, 11.11s/it][2022-12-20 19:04:37,237] [INFO] [timer.py:197:stop] 0/5166, RunningAvgSamplesPerSec=5.872403584942668, CurrSamplesPerSec=5.296938777725091, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2552/5000 [8:42:12<7:32:21, 11.09s/it][2022-12-20 19:04:48,203] [INFO] [timer.py:197:stop] 0/5168, RunningAvgSamplesPerSec=5.872415469603348, CurrSamplesPerSec=5.380284910590834, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2553/5000 [8:42:23<7:30:41, 11.05s/it][2022-12-20 19:04:59,223] [INFO] [timer.py:197:stop] 0/5170, RunningAvgSamplesPerSec=5.872416055550142, CurrSamplesPerSec=5.319675542653149, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2554/5000 [8:42:34<7:30:07, 11.04s/it][2022-12-20 19:05:10,222] [INFO] [timer.py:197:stop] 0/5172, RunningAvgSamplesPerSec=5.872420990808128, CurrSamplesPerSec=5.322553849139647, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2555/5000 [8:42:45<7:29:25, 11.03s/it][2022-12-20 19:05:21,240] [INFO] [timer.py:197:stop] 0/5174, RunningAvgSamplesPerSec=5.8724217157010745, CurrSamplesPerSec=5.329474805555454, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2556/5000 [8:42:56<7:29:06, 11.03s/it][2022-12-20 19:05:32,257] [INFO] [timer.py:197:stop] 0/5176, RunningAvgSamplesPerSec=5.8724226704020595, CurrSamplesPerSec=5.335125783622286, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2557/5000 [8:43:07<7:28:49, 11.02s/it][2022-12-20 19:05:43,208] [INFO] [timer.py:197:stop] 0/5178, RunningAvgSamplesPerSec=5.872437516115028, CurrSamplesPerSec=5.348096486521649, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2558/5000 [8:43:18<7:27:45, 11.00s/it][2022-12-20 19:05:54,247] [INFO] [logging.py:68:log_dist] [Rank 0] step=2590, skipped=3, lr=[5.364444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:05:54,249] [INFO] [timer.py:197:stop] 0/5180, RunningAvgSamplesPerSec=5.8724337866150425, CurrSamplesPerSec=5.324551337862783, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2559/5000 [8:43:29<7:28:03, 11.01s/it][2022-12-20 19:06:05,223] [INFO] [timer.py:197:stop] 0/5182, RunningAvgSamplesPerSec=5.872443655345555, CurrSamplesPerSec=5.3557300438118265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2560/5000 [8:43:40<7:27:24, 11.00s/it][2022-12-20 19:06:16,268] [INFO] [timer.py:197:stop] 0/5184, RunningAvgSamplesPerSec=5.872438911687643, CurrSamplesPerSec=5.328012693444929, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2561/5000 [8:43:51<7:27:44, 11.01s/it][2022-12-20 19:06:27,264] [INFO] [timer.py:197:stop] 0/5186, RunningAvgSamplesPerSec=5.87244440592142, CurrSamplesPerSec=5.323439865013492, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████     | 2562/5000 [8:44:02<7:27:19, 11.01s/it][2022-12-20 19:06:38,231] [INFO] [timer.py:197:stop] 0/5188, RunningAvgSamplesPerSec=5.872455876510898, CurrSamplesPerSec=5.331492529766117, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2563/5000 [8:44:13<7:26:38, 11.00s/it][2022-12-20 19:06:49,232] [INFO] [timer.py:197:stop] 0/5190, RunningAvgSamplesPerSec=5.872460355966199, CurrSamplesPerSec=5.335391096511791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2564/5000 [8:44:24<7:26:30, 11.00s/it][2022-12-20 19:07:00,215] [INFO] [timer.py:197:stop] 0/5192, RunningAvgSamplesPerSec=5.87246847264832, CurrSamplesPerSec=5.361601994859694, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2565/5000 [8:44:35<7:26:08, 10.99s/it][2022-12-20 19:07:11,199] [INFO] [timer.py:197:stop] 0/5194, RunningAvgSamplesPerSec=5.8724761275646, CurrSamplesPerSec=5.316681352290014, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2566/5000 [8:44:46<7:25:51, 10.99s/it][2022-12-20 19:07:22,196] [INFO] [timer.py:197:stop] 0/5196, RunningAvgSamplesPerSec=5.8724814691814355, CurrSamplesPerSec=5.35660875508661, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2567/5000 [8:44:57<7:25:44, 10.99s/it][2022-12-20 19:07:33,216] [INFO] [timer.py:197:stop] 0/5198, RunningAvgSamplesPerSec=5.87248225048813, CurrSamplesPerSec=5.334542018516444, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2568/5000 [8:45:08<7:25:53, 11.00s/it][2022-12-20 19:07:44,242] [INFO] [logging.py:68:log_dist] [Rank 0] step=2600, skipped=3, lr=[5.342222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:07:44,243] [INFO] [timer.py:197:stop] 0/5200, RunningAvgSamplesPerSec=5.8724811897992995, CurrSamplesPerSec=5.299574265855183, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2569/5000 [8:45:19<7:26:02, 11.01s/it][2022-12-20 19:07:55,241] [INFO] [timer.py:197:stop] 0/5202, RunningAvgSamplesPerSec=5.872486257874632, CurrSamplesPerSec=5.350744317000331, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2570/5000 [8:45:30<7:25:43, 11.01s/it][2022-12-20 19:08:06,215] [INFO] [timer.py:197:stop] 0/5204, RunningAvgSamplesPerSec=5.872496423513133, CurrSamplesPerSec=5.332045336421326, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2571/5000 [8:45:41<7:25:08, 11.00s/it][2022-12-20 19:08:17,197] [INFO] [timer.py:197:stop] 0/5206, RunningAvgSamplesPerSec=5.872504351093058, CurrSamplesPerSec=5.355481509111017, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2572/5000 [8:45:52<7:24:48, 10.99s/it][2022-12-20 19:08:28,194] [INFO] [timer.py:197:stop] 0/5208, RunningAvgSamplesPerSec=5.872509629874759, CurrSamplesPerSec=5.308374349478798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2573/5000 [8:46:03<7:24:41, 10.99s/it][2022-12-20 19:08:39,237] [INFO] [timer.py:197:stop] 0/5210, RunningAvgSamplesPerSec=5.872505283142606, CurrSamplesPerSec=5.310614817538794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 51%|█████▏    | 2574/5000 [8:46:14<7:25:05, 11.01s/it][2022-12-20 19:08:50,202] [INFO] [timer.py:197:stop] 0/5212, RunningAvgSamplesPerSec=5.872516952704419, CurrSamplesPerSec=5.33588764820488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2575/5000 [8:46:25<7:24:23, 11.00s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 5.328888888888889e-06, 'epoch': 62.8}

 52%|█████▏    | 2575/5000 [8:46:25<7:24:23, 11.00s/it][2022-12-20 19:09:01,234] [INFO] [timer.py:197:stop] 0/5214, RunningAvgSamplesPerSec=5.87251514521287, CurrSamplesPerSec=5.318851271947023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2576/5000 [8:46:36<7:24:39, 11.01s/it][2022-12-20 19:09:12,298] [INFO] [timer.py:197:stop] 0/5216, RunningAvgSamplesPerSec=5.8725063446183485, CurrSamplesPerSec=5.322440294925763, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2577/5000 [8:46:47<7:25:10, 11.02s/it][2022-12-20 19:09:23,330] [INFO] [timer.py:197:stop] 0/5218, RunningAvgSamplesPerSec=5.872504532882617, CurrSamplesPerSec=5.313965497677407, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2578/5000 [8:46:58<7:25:05, 11.03s/it][2022-12-20 19:09:34,319] [INFO] [logging.py:68:log_dist] [Rank 0] step=2610, skipped=3, lr=[5.320000000000001e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:09:34,320] [INFO] [timer.py:197:stop] 0/5220, RunningAvgSamplesPerSec=5.8725111269640085, CurrSamplesPerSec=5.316674402290827, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2579/5000 [8:47:09<7:24:28, 11.02s/it][2022-12-20 19:09:45,320] [INFO] [timer.py:197:stop] 0/5222, RunningAvgSamplesPerSec=5.87251572552731, CurrSamplesPerSec=5.358455821953807, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2580/5000 [8:47:20<7:24:05, 11.01s/it][2022-12-20 19:09:56,348] [INFO] [timer.py:197:stop] 0/5224, RunningAvgSamplesPerSec=5.872514471085712, CurrSamplesPerSec=5.319845909731444, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2581/5000 [8:47:31<7:24:07, 11.02s/it][2022-12-20 19:10:07,296] [INFO] [timer.py:197:stop] 0/5226, RunningAvgSamplesPerSec=5.872529894701253, CurrSamplesPerSec=5.358852475464828, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2582/5000 [8:47:42<7:23:07, 11.00s/it][2022-12-20 19:10:18,297] [INFO] [timer.py:197:stop] 0/5228, RunningAvgSamplesPerSec=5.8725341969919205, CurrSamplesPerSec=5.340574758548974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2583/5000 [8:47:53<7:23:00, 11.00s/it][2022-12-20 19:10:28,414] [INFO] [timer.py:197:stop] 0/5230, RunningAvgSamplesPerSec=5.872718591619811, CurrSamplesPerSec=5.321307761866535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2584/5000 [8:48:08<8:11:53, 12.22s/it][2022-12-20 19:10:39,413] [INFO] [timer.py:197:stop] 0/5232, RunningAvgSamplesPerSec=5.872723067245804, CurrSamplesPerSec=5.339408156646932, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2585/5000 [8:48:19<7:56:52, 11.85s/it][2022-12-20 19:10:50,388] [INFO] [timer.py:197:stop] 0/5234, RunningAvgSamplesPerSec=5.8727324689966585, CurrSamplesPerSec=5.352125238820002, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2586/5000 [8:48:30<7:46:27, 11.59s/it][2022-12-20 19:11:01,435] [INFO] [timer.py:197:stop] 0/5236, RunningAvgSamplesPerSec=5.872727255673875, CurrSamplesPerSec=5.312134462998311, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2587/5000 [8:48:41<7:39:45, 11.43s/it][2022-12-20 19:11:12,486] [INFO] [timer.py:197:stop] 0/5238, RunningAvgSamplesPerSec=5.872721115861741, CurrSamplesPerSec=5.314858548002951, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2588/5000 [8:48:53<7:34:45, 11.31s/it][2022-12-20 19:11:23,474] [INFO] [logging.py:68:log_dist] [Rank 0] step=2620, skipped=3, lr=[5.297777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:11:23,475] [INFO] [timer.py:197:stop] 0/5240, RunningAvgSamplesPerSec=5.872727751635108, CurrSamplesPerSec=5.35398736801392, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2589/5000 [8:49:04<7:30:46, 11.22s/it][2022-12-20 19:11:34,484] [INFO] [timer.py:197:stop] 0/5242, RunningAvgSamplesPerSec=5.872730459308229, CurrSamplesPerSec=5.344973801738098, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2590/5000 [8:49:14<7:27:29, 11.14s/it][2022-12-20 19:11:45,492] [INFO] [timer.py:197:stop] 0/5244, RunningAvgSamplesPerSec=5.872733275261813, CurrSamplesPerSec=5.302214891401318, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2591/5000 [8:49:26<7:26:24, 11.12s/it][2022-12-20 19:11:56,550] [INFO] [timer.py:197:stop] 0/5246, RunningAvgSamplesPerSec=5.872725738328682, CurrSamplesPerSec=5.3096190072228975, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2592/5000 [8:49:37<7:25:14, 11.09s/it][2022-12-20 19:12:07,540] [INFO] [timer.py:197:stop] 0/5248, RunningAvgSamplesPerSec=5.872732196179646, CurrSamplesPerSec=5.351492512348972, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2593/5000 [8:49:48<7:23:21, 11.05s/it][2022-12-20 19:12:18,566] [INFO] [timer.py:197:stop] 0/5250, RunningAvgSamplesPerSec=5.872731112776292, CurrSamplesPerSec=5.286961695310178, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2594/5000 [8:49:59<7:23:28, 11.06s/it][2022-12-20 19:12:29,613] [INFO] [timer.py:197:stop] 0/5252, RunningAvgSamplesPerSec=5.872725859377988, CurrSamplesPerSec=5.312414105216711, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2595/5000 [8:50:10<7:23:11, 11.06s/it][2022-12-20 19:12:40,654] [INFO] [timer.py:197:stop] 0/5254, RunningAvgSamplesPerSec=5.8727218447482805, CurrSamplesPerSec=5.322010183681454, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2596/5000 [8:50:21<7:22:45, 11.05s/it][2022-12-20 19:12:51,641] [INFO] [timer.py:197:stop] 0/5256, RunningAvgSamplesPerSec=5.8727288260653765, CurrSamplesPerSec=5.364719056554497, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2597/5000 [8:50:32<7:22:07, 11.04s/it][2022-12-20 19:13:02,699] [INFO] [timer.py:197:stop] 0/5258, RunningAvgSamplesPerSec=5.872721269421697, CurrSamplesPerSec=5.325807820713338, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2598/5000 [8:50:43<7:21:46, 11.04s/it][2022-12-20 19:13:13,727] [INFO] [logging.py:68:log_dist] [Rank 0] step=2630, skipped=3, lr=[5.275555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:13:13,729] [INFO] [timer.py:197:stop] 0/5260, RunningAvgSamplesPerSec=5.872719315890642, CurrSamplesPerSec=5.320843872690054, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2599/5000 [8:50:54<7:20:56, 11.02s/it][2022-12-20 19:13:24,684] [INFO] [timer.py:197:stop] 0/5262, RunningAvgSamplesPerSec=5.872732633905665, CurrSamplesPerSec=5.3437231530299885, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2600/5000 [8:51:05<7:19:59, 11.00s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 5.273333333333333e-06, 'epoch': 63.41}

 52%|█████▏    | 2600/5000 [8:51:05<7:19:59, 11.00s/it][2022-12-20 19:13:35,609] [INFO] [timer.py:197:stop] 0/5264, RunningAvgSamplesPerSec=5.872752614533249, CurrSamplesPerSec=5.3727328670988275, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2601/5000 [8:51:16<7:18:54, 10.98s/it][2022-12-20 19:13:46,582] [INFO] [timer.py:197:stop] 0/5266, RunningAvgSamplesPerSec=5.872762270962602, CurrSamplesPerSec=5.327367892773217, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2602/5000 [8:51:27<7:18:38, 10.98s/it][2022-12-20 19:13:57,509] [INFO] [timer.py:197:stop] 0/5268, RunningAvgSamplesPerSec=5.8727816184647095, CurrSamplesPerSec=5.3667060506336615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2603/5000 [8:51:38<7:18:11, 10.97s/it][2022-12-20 19:14:08,523] [INFO] [timer.py:197:stop] 0/5270, RunningAvgSamplesPerSec=5.872783312205159, CurrSamplesPerSec=5.313508146566945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2604/5000 [8:51:49<7:18:27, 10.98s/it][2022-12-20 19:14:19,507] [INFO] [timer.py:197:stop] 0/5272, RunningAvgSamplesPerSec=5.872791127127035, CurrSamplesPerSec=5.332071391057764, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2605/5000 [8:52:00<7:18:46, 10.99s/it][2022-12-20 19:14:30,545] [INFO] [timer.py:197:stop] 0/5274, RunningAvgSamplesPerSec=5.872787866177656, CurrSamplesPerSec=5.318367158356488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2606/5000 [8:52:11<7:18:43, 11.00s/it][2022-12-20 19:14:41,566] [INFO] [timer.py:197:stop] 0/5276, RunningAvgSamplesPerSec=5.87278749320739, CurrSamplesPerSec=5.300960930956442, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2607/5000 [8:52:22<7:19:07, 11.01s/it][2022-12-20 19:14:52,597] [INFO] [timer.py:197:stop] 0/5278, RunningAvgSamplesPerSec=5.872785592058974, CurrSamplesPerSec=5.3137071498402895, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2608/5000 [8:52:33<7:19:23, 11.02s/it][2022-12-20 19:15:03,614] [INFO] [logging.py:68:log_dist] [Rank 0] step=2640, skipped=3, lr=[5.2533333333333336e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:15:03,615] [INFO] [timer.py:197:stop] 0/5280, RunningAvgSamplesPerSec=5.872786121908367, CurrSamplesPerSec=5.339306201465275, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2609/5000 [8:52:44<7:18:59, 11.02s/it][2022-12-20 19:15:14,644] [INFO] [timer.py:197:stop] 0/5282, RunningAvgSamplesPerSec=5.8727846037916835, CurrSamplesPerSec=5.317423000553502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2610/5000 [8:52:55<7:18:51, 11.02s/it][2022-12-20 19:15:25,607] [INFO] [timer.py:197:stop] 0/5284, RunningAvgSamplesPerSec=5.872796393456776, CurrSamplesPerSec=5.368255609766087, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2611/5000 [8:53:06<7:17:50, 11.00s/it][2022-12-20 19:15:36,616] [INFO] [timer.py:197:stop] 0/5286, RunningAvgSamplesPerSec=5.872798888387285, CurrSamplesPerSec=5.314350541477422, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2612/5000 [8:53:17<7:18:15, 11.01s/it][2022-12-20 19:15:47,606] [INFO] [timer.py:197:stop] 0/5288, RunningAvgSamplesPerSec=5.872805041662639, CurrSamplesPerSec=5.364112719093264, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2613/5000 [8:53:28<7:17:30, 11.00s/it][2022-12-20 19:15:58,593] [INFO] [timer.py:197:stop] 0/5290, RunningAvgSamplesPerSec=5.872811978991787, CurrSamplesPerSec=5.343599971223219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2614/5000 [8:53:39<7:16:50, 10.99s/it][2022-12-20 19:16:09,580] [INFO] [timer.py:197:stop] 0/5292, RunningAvgSamplesPerSec=5.872818754723418, CurrSamplesPerSec=5.316514978510569, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2615/5000 [8:53:50<7:17:30, 11.01s/it][2022-12-20 19:16:20,629] [INFO] [timer.py:197:stop] 0/5294, RunningAvgSamplesPerSec=5.872813005560405, CurrSamplesPerSec=5.322860343063403, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2616/5000 [8:54:01<7:17:13, 11.00s/it][2022-12-20 19:16:31,603] [INFO] [timer.py:197:stop] 0/5296, RunningAvgSamplesPerSec=5.872822677605095, CurrSamplesPerSec=5.345316518227611, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2617/5000 [8:54:12<7:16:55, 11.00s/it][2022-12-20 19:16:42,569] [INFO] [timer.py:197:stop] 0/5298, RunningAvgSamplesPerSec=5.872833789152779, CurrSamplesPerSec=5.369290721570961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2618/5000 [8:54:23<7:16:29, 10.99s/it][2022-12-20 19:16:53,592] [INFO] [logging.py:68:log_dist] [Rank 0] step=2650, skipped=3, lr=[5.231111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:16:53,593] [INFO] [timer.py:197:stop] 0/5300, RunningAvgSamplesPerSec=5.872833023221077, CurrSamplesPerSec=5.32964199145988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2619/5000 [8:54:34<7:16:07, 10.99s/it][2022-12-20 19:17:04,583] [INFO] [timer.py:197:stop] 0/5302, RunningAvgSamplesPerSec=5.872839567170281, CurrSamplesPerSec=5.321767089725632, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2620/5000 [8:54:45<7:16:04, 10.99s/it][2022-12-20 19:17:15,561] [INFO] [timer.py:197:stop] 0/5304, RunningAvgSamplesPerSec=5.872848409600101, CurrSamplesPerSec=5.342748278693963, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2621/5000 [8:54:56<7:15:54, 10.99s/it][2022-12-20 19:17:26,564] [INFO] [timer.py:197:stop] 0/5306, RunningAvgSamplesPerSec=5.87285184991018, CurrSamplesPerSec=5.3343825814635855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2622/5000 [8:55:07<7:16:09, 11.00s/it][2022-12-20 19:17:37,613] [INFO] [timer.py:197:stop] 0/5308, RunningAvgSamplesPerSec=5.87284632523787, CurrSamplesPerSec=5.3181749705221035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2623/5000 [8:55:18<7:16:17, 11.01s/it][2022-12-20 19:17:48,596] [INFO] [timer.py:197:stop] 0/5310, RunningAvgSamplesPerSec=5.872853908669255, CurrSamplesPerSec=5.3613664070235965, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▏    | 2624/5000 [8:55:29<7:16:02, 11.01s/it][2022-12-20 19:17:58,745] [INFO] [timer.py:197:stop] 0/5312, RunningAvgSamplesPerSec=5.873028763706414, CurrSamplesPerSec=6.2490212401411664, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 19:18:09,793] [INFO] [timer.py:197:stop] 0/5314, RunningAvgSamplesPerSec=5.873022553566531, CurrSamplesPerSec=5.292588362031317, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 52%|█████▎    | 2625/5000 [8:55:45<8:17:44, 12.57s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 5.215555555555556e-06, 'epoch': 64.02}

 52%|█████▎    | 2625/5000 [8:55:45<8:17:44, 12.57s/it][2022-12-20 19:18:20,859] [INFO] [timer.py:197:stop] 0/5316, RunningAvgSamplesPerSec=5.873013301741487, CurrSamplesPerSec=5.29528491123225, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2626/5000 [8:55:56<7:59:37, 12.12s/it][2022-12-20 19:18:31,797] [INFO] [timer.py:197:stop] 0/5318, RunningAvgSamplesPerSec=5.873030148960379, CurrSamplesPerSec=5.369022456150541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2627/5000 [8:56:07<7:45:22, 11.77s/it][2022-12-20 19:18:42,872] [INFO] [logging.py:68:log_dist] [Rank 0] step=2660, skipped=3, lr=[5.208888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:18:42,874] [INFO] [timer.py:197:stop] 0/5320, RunningAvgSamplesPerSec=5.873018633455563, CurrSamplesPerSec=5.24457424332001, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2628/5000 [8:56:18<7:36:59, 11.56s/it][2022-12-20 19:18:53,901] [INFO] [timer.py:197:stop] 0/5322, RunningAvgSamplesPerSec=5.873017277272148, CurrSamplesPerSec=5.3174409071449436, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2629/5000 [8:56:29<7:30:29, 11.40s/it][2022-12-20 19:19:04,963] [INFO] [timer.py:197:stop] 0/5324, RunningAvgSamplesPerSec=5.873008760019579, CurrSamplesPerSec=5.296574855184251, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2630/5000 [8:56:40<7:26:18, 11.30s/it][2022-12-20 19:19:16,039] [INFO] [timer.py:197:stop] 0/5326, RunningAvgSamplesPerSec=5.872997820805419, CurrSamplesPerSec=5.304639472268364, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2631/5000 [8:56:51<7:23:27, 11.23s/it][2022-12-20 19:19:27,078] [INFO] [timer.py:197:stop] 0/5328, RunningAvgSamplesPerSec=5.872994519269735, CurrSamplesPerSec=5.322823612591185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2632/5000 [8:57:02<7:20:59, 11.17s/it][2022-12-20 19:19:38,113] [INFO] [timer.py:197:stop] 0/5330, RunningAvgSamplesPerSec=5.8729917629827995, CurrSamplesPerSec=5.332183450079617, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2633/5000 [8:57:13<7:19:10, 11.13s/it][2022-12-20 19:19:49,105] [INFO] [timer.py:197:stop] 0/5332, RunningAvgSamplesPerSec=5.87299790650301, CurrSamplesPerSec=5.298354809692404, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2634/5000 [8:57:24<7:17:19, 11.09s/it][2022-12-20 19:20:00,116] [INFO] [timer.py:197:stop] 0/5334, RunningAvgSamplesPerSec=5.872999848092622, CurrSamplesPerSec=5.341615165890961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2635/5000 [8:57:35<7:16:12, 11.07s/it][2022-12-20 19:20:11,157] [INFO] [timer.py:197:stop] 0/5336, RunningAvgSamplesPerSec=5.8729957919291245, CurrSamplesPerSec=5.308615381536392, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2636/5000 [8:57:46<7:15:43, 11.06s/it][2022-12-20 19:20:22,228] [INFO] [timer.py:197:stop] 0/5338, RunningAvgSamplesPerSec=5.8729860320114735, CurrSamplesPerSec=5.309025060272417, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2637/5000 [8:57:57<7:15:40, 11.06s/it][2022-12-20 19:20:33,212] [INFO] [logging.py:68:log_dist] [Rank 0] step=2670, skipped=3, lr=[5.186666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:20:33,214] [INFO] [timer.py:197:stop] 0/5340, RunningAvgSamplesPerSec=5.872993424695448, CurrSamplesPerSec=5.3231822841240914, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2638/5000 [8:58:08<7:14:35, 11.04s/it][2022-12-20 19:20:44,225] [INFO] [timer.py:197:stop] 0/5342, RunningAvgSamplesPerSec=5.872995579110673, CurrSamplesPerSec=5.313117966714269, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2639/5000 [8:58:19<7:14:04, 11.03s/it][2022-12-20 19:20:55,207] [INFO] [timer.py:197:stop] 0/5344, RunningAvgSamplesPerSec=5.873003614027115, CurrSamplesPerSec=5.349213163734925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2640/5000 [8:58:30<7:13:18, 11.02s/it][2022-12-20 19:21:06,238] [INFO] [timer.py:197:stop] 0/5346, RunningAvgSamplesPerSec=5.873001452918266, CurrSamplesPerSec=5.304556241183625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2641/5000 [8:58:41<7:13:17, 11.02s/it][2022-12-20 19:21:17,291] [INFO] [timer.py:197:stop] 0/5348, RunningAvgSamplesPerSec=5.872994980954503, CurrSamplesPerSec=5.302913328929573, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2642/5000 [8:58:52<7:13:29, 11.03s/it][2022-12-20 19:21:28,334] [INFO] [timer.py:197:stop] 0/5350, RunningAvgSamplesPerSec=5.872990571074898, CurrSamplesPerSec=5.312529124508324, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2643/5000 [8:59:03<7:13:27, 11.03s/it][2022-12-20 19:21:39,400] [INFO] [timer.py:197:stop] 0/5352, RunningAvgSamplesPerSec=5.8729814620583305, CurrSamplesPerSec=5.300411201504138, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2644/5000 [8:59:14<7:13:38, 11.04s/it][2022-12-20 19:21:50,400] [INFO] [timer.py:197:stop] 0/5354, RunningAvgSamplesPerSec=5.872985804174806, CurrSamplesPerSec=5.3324870287103385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2645/5000 [8:59:25<7:12:56, 11.03s/it][2022-12-20 19:22:01,447] [INFO] [timer.py:197:stop] 0/5356, RunningAvgSamplesPerSec=5.872980512400144, CurrSamplesPerSec=5.290296556057971, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2646/5000 [8:59:37<7:12:57, 11.04s/it][2022-12-20 19:22:12,424] [INFO] [timer.py:197:stop] 0/5358, RunningAvgSamplesPerSec=5.872989312307629, CurrSamplesPerSec=5.353349290216511, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2647/5000 [8:59:48<7:12:05, 11.02s/it][2022-12-20 19:22:23,405] [INFO] [logging.py:68:log_dist] [Rank 0] step=2680, skipped=3, lr=[5.164444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:22:23,407] [INFO] [timer.py:197:stop] 0/5360, RunningAvgSamplesPerSec=5.872997299604377, CurrSamplesPerSec=5.3312633921859645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2648/5000 [8:59:59<7:11:29, 11.01s/it][2022-12-20 19:22:34,445] [INFO] [timer.py:197:stop] 0/5362, RunningAvgSamplesPerSec=5.872993868370962, CurrSamplesPerSec=5.305449906368411, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2649/5000 [9:00:10<7:11:40, 11.02s/it][2022-12-20 19:22:45,443] [INFO] [timer.py:197:stop] 0/5364, RunningAvgSamplesPerSec=5.87299853236085, CurrSamplesPerSec=5.331731853442185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2650/5000 [9:00:21<7:11:16, 11.01s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 5.1600000000000006e-06, 'epoch': 64.63}

 53%|█████▎    | 2650/5000 [9:00:21<7:11:16, 11.01s/it][2022-12-20 19:22:56,439] [INFO] [timer.py:197:stop] 0/5366, RunningAvgSamplesPerSec=5.873003617885527, CurrSamplesPerSec=5.319256629427013, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2651/5000 [9:00:32<7:10:54, 11.01s/it][2022-12-20 19:23:07,423] [INFO] [timer.py:197:stop] 0/5368, RunningAvgSamplesPerSec=5.873011208773891, CurrSamplesPerSec=5.34459665153082, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2652/5000 [9:00:43<7:10:27, 11.00s/it][2022-12-20 19:23:18,492] [INFO] [timer.py:197:stop] 0/5370, RunningAvgSamplesPerSec=5.873001512588647, CurrSamplesPerSec=5.290541162699469, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2653/5000 [9:00:54<7:11:05, 11.02s/it][2022-12-20 19:23:29,543] [INFO] [timer.py:197:stop] 0/5372, RunningAvgSamplesPerSec=5.872995480082908, CurrSamplesPerSec=5.304801958612517, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2654/5000 [9:01:05<7:11:15, 11.03s/it][2022-12-20 19:23:40,531] [INFO] [timer.py:197:stop] 0/5374, RunningAvgSamplesPerSec=5.873001862037056, CurrSamplesPerSec=5.329954593752459, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2655/5000 [9:01:16<7:10:35, 11.02s/it][2022-12-20 19:23:51,565] [INFO] [timer.py:197:stop] 0/5376, RunningAvgSamplesPerSec=5.872999266143301, CurrSamplesPerSec=5.328287664122435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2656/5000 [9:01:27<7:10:35, 11.02s/it][2022-12-20 19:24:02,611] [INFO] [timer.py:197:stop] 0/5378, RunningAvgSamplesPerSec=5.872993929326478, CurrSamplesPerSec=5.3147495309620485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2657/5000 [9:01:38<7:10:41, 11.03s/it][2022-12-20 19:24:13,686] [INFO] [logging.py:68:log_dist] [Rank 0] step=2690, skipped=3, lr=[5.1422222222222225e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:24:13,688] [INFO] [timer.py:197:stop] 0/5380, RunningAvgSamplesPerSec=5.87298256364583, CurrSamplesPerSec=5.278240500087736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2658/5000 [9:01:49<7:11:03, 11.04s/it][2022-12-20 19:24:24,677] [INFO] [timer.py:197:stop] 0/5382, RunningAvgSamplesPerSec=5.872988673437353, CurrSamplesPerSec=5.343458074523034, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2659/5000 [9:02:00<7:10:15, 11.03s/it][2022-12-20 19:24:35,634] [INFO] [timer.py:197:stop] 0/5384, RunningAvgSamplesPerSec=5.873001617676269, CurrSamplesPerSec=5.353229507427144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2660/5000 [9:02:11<7:09:14, 11.01s/it][2022-12-20 19:24:46,706] [INFO] [timer.py:197:stop] 0/5386, RunningAvgSamplesPerSec=5.872991035787454, CurrSamplesPerSec=5.290466923339986, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2661/5000 [9:02:22<7:09:49, 11.03s/it][2022-12-20 19:24:57,775] [INFO] [timer.py:197:stop] 0/5388, RunningAvgSamplesPerSec=5.872981542600957, CurrSamplesPerSec=5.292540569733379, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2662/5000 [9:02:33<7:10:08, 11.04s/it][2022-12-20 19:25:08,835] [INFO] [timer.py:197:stop] 0/5390, RunningAvgSamplesPerSec=5.872973786269931, CurrSamplesPerSec=5.308869244711767, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2663/5000 [9:02:44<7:10:12, 11.05s/it][2022-12-20 19:25:19,883] [INFO] [timer.py:197:stop] 0/5392, RunningAvgSamplesPerSec=5.872968507954164, CurrSamplesPerSec=5.317162209985033, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2664/5000 [9:02:55<7:10:03, 11.05s/it][2022-12-20 19:25:30,920] [INFO] [timer.py:197:stop] 0/5394, RunningAvgSamplesPerSec=5.8729655928815925, CurrSamplesPerSec=5.3190289639543495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2665/5000 [9:03:06<7:09:45, 11.04s/it][2022-12-20 19:25:41,046] [INFO] [timer.py:197:stop] 0/5396, RunningAvgSamplesPerSec=5.873142360979769, CurrSamplesPerSec=5.3161761557581215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2666/5000 [9:03:21<7:56:42, 12.25s/it][2022-12-20 19:25:52,015] [INFO] [timer.py:197:stop] 0/5398, RunningAvgSamplesPerSec=5.873152696212998, CurrSamplesPerSec=5.3781859138350505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2667/5000 [9:03:32<7:40:53, 11.85s/it][2022-12-20 19:26:02,981] [INFO] [logging.py:68:log_dist] [Rank 0] step=2700, skipped=3, lr=[5.12e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:26:02,982] [INFO] [timer.py:197:stop] 0/5400, RunningAvgSamplesPerSec=5.873163369617966, CurrSamplesPerSec=5.333012706111894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2668/5000 [9:03:43<7:30:40, 11.60s/it][2022-12-20 19:26:13,998] [INFO] [timer.py:197:stop] 0/5402, RunningAvgSamplesPerSec=5.873164243362551, CurrSamplesPerSec=5.313742702734375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2669/5000 [9:03:54<7:23:41, 11.42s/it][2022-12-20 19:26:24,956] [INFO] [timer.py:197:stop] 0/5404, RunningAvgSamplesPerSec=5.873176910215417, CurrSamplesPerSec=5.362685322830571, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2670/5000 [9:04:05<7:17:53, 11.28s/it][2022-12-20 19:26:35,948] [INFO] [timer.py:197:stop] 0/5406, RunningAvgSamplesPerSec=5.873182388507867, CurrSamplesPerSec=5.3148699129711945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2671/5000 [9:04:16<7:15:00, 11.21s/it][2022-12-20 19:26:46,990] [INFO] [timer.py:197:stop] 0/5408, RunningAvgSamplesPerSec=5.873177998257727, CurrSamplesPerSec=5.316868798515467, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2672/5000 [9:04:27<7:12:28, 11.15s/it][2022-12-20 19:26:57,989] [INFO] [timer.py:197:stop] 0/5410, RunningAvgSamplesPerSec=5.873182089474428, CurrSamplesPerSec=5.322459712775036, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2673/5000 [9:04:38<7:10:45, 11.11s/it][2022-12-20 19:27:08,981] [INFO] [timer.py:197:stop] 0/5412, RunningAvgSamplesPerSec=5.87318763531925, CurrSamplesPerSec=5.342952668647412, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 53%|█████▎    | 2674/5000 [9:04:49<7:09:02, 11.07s/it][2022-12-20 19:27:19,959] [INFO] [timer.py:197:stop] 0/5414, RunningAvgSamplesPerSec=5.873195970017013, CurrSamplesPerSec=5.340773244206991, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2675/5000 [9:05:00<7:08:15, 11.05s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 5.1044444444444455e-06, 'epoch': 65.24}

 54%|█████▎    | 2675/5000 [9:05:00<7:08:15, 11.05s/it][2022-12-20 19:27:31,007] [INFO] [timer.py:197:stop] 0/5416, RunningAvgSamplesPerSec=5.873190868120821, CurrSamplesPerSec=5.314474272314122, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2676/5000 [9:05:11<7:08:03, 11.05s/it][2022-12-20 19:27:42,050] [INFO] [timer.py:197:stop] 0/5418, RunningAvgSamplesPerSec=5.87318633516062, CurrSamplesPerSec=5.318414785998138, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2677/5000 [9:05:22<7:07:48, 11.05s/it][2022-12-20 19:27:53,120] [INFO] [logging.py:68:log_dist] [Rank 0] step=2710, skipped=3, lr=[5.0977777777777786e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:27:53,122] [INFO] [timer.py:197:stop] 0/5420, RunningAvgSamplesPerSec=5.873175921260329, CurrSamplesPerSec=5.294825547890776, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2678/5000 [9:05:33<7:07:23, 11.04s/it][2022-12-20 19:28:04,091] [INFO] [timer.py:197:stop] 0/5422, RunningAvgSamplesPerSec=5.873185786736362, CurrSamplesPerSec=5.347557178434134, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2679/5000 [9:05:44<7:06:40, 11.03s/it][2022-12-20 19:28:15,101] [INFO] [timer.py:197:stop] 0/5424, RunningAvgSamplesPerSec=5.8731878712614325, CurrSamplesPerSec=5.337774368911436, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2680/5000 [9:05:55<7:06:00, 11.02s/it][2022-12-20 19:28:26,061] [INFO] [timer.py:197:stop] 0/5426, RunningAvgSamplesPerSec=5.873199868641912, CurrSamplesPerSec=5.362978670031498, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2681/5000 [9:06:06<7:04:47, 10.99s/it][2022-12-20 19:28:37,043] [INFO] [timer.py:197:stop] 0/5428, RunningAvgSamplesPerSec=5.873207361822999, CurrSamplesPerSec=5.31523930089995, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2682/5000 [9:06:17<7:04:34, 10.99s/it][2022-12-20 19:28:48,008] [INFO] [timer.py:197:stop] 0/5430, RunningAvgSamplesPerSec=5.873218476979349, CurrSamplesPerSec=5.335861132002502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2683/5000 [9:06:28<7:04:18, 10.99s/it][2022-12-20 19:28:58,974] [INFO] [timer.py:197:stop] 0/5432, RunningAvgSamplesPerSec=5.873229123034713, CurrSamplesPerSec=5.3508159913250095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2684/5000 [9:06:39<7:03:36, 10.97s/it][2022-12-20 19:29:09,964] [INFO] [timer.py:197:stop] 0/5434, RunningAvgSamplesPerSec=5.873234898687209, CurrSamplesPerSec=5.307958053845484, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2685/5000 [9:06:50<7:04:04, 10.99s/it][2022-12-20 19:29:20,944] [INFO] [timer.py:197:stop] 0/5436, RunningAvgSamplesPerSec=5.873242852156995, CurrSamplesPerSec=5.353249577609812, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2686/5000 [9:07:01<7:03:44, 10.99s/it][2022-12-20 19:29:31,933] [INFO] [timer.py:197:stop] 0/5438, RunningAvgSamplesPerSec=5.873249210420044, CurrSamplesPerSec=5.344848220984026, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▎    | 2687/5000 [9:07:12<7:03:36, 10.99s/it][2022-12-20 19:29:42,926] [INFO] [logging.py:68:log_dist] [Rank 0] step=2720, skipped=3, lr=[5.075555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:29:42,927] [INFO] [timer.py:197:stop] 0/5440, RunningAvgSamplesPerSec=5.873254239134911, CurrSamplesPerSec=5.3420050776574, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2688/5000 [9:07:23<7:03:37, 10.99s/it][2022-12-20 19:29:53,919] [INFO] [timer.py:197:stop] 0/5442, RunningAvgSamplesPerSec=5.8732595932174725, CurrSamplesPerSec=5.35391048311184, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2689/5000 [9:07:34<7:03:16, 10.99s/it][2022-12-20 19:30:04,920] [INFO] [timer.py:197:stop] 0/5444, RunningAvgSamplesPerSec=5.873263264122879, CurrSamplesPerSec=5.33441311122531, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2690/5000 [9:07:45<7:02:46, 10.98s/it][2022-12-20 19:30:15,888] [INFO] [timer.py:197:stop] 0/5446, RunningAvgSamplesPerSec=5.873273570455514, CurrSamplesPerSec=5.32995946191834, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2691/5000 [9:07:56<7:02:59, 10.99s/it][2022-12-20 19:30:26,921] [INFO] [timer.py:197:stop] 0/5448, RunningAvgSamplesPerSec=5.873270970160009, CurrSamplesPerSec=5.314747215976855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2692/5000 [9:08:07<7:02:38, 10.99s/it][2022-12-20 19:30:37,853] [INFO] [timer.py:197:stop] 0/5450, RunningAvgSamplesPerSec=5.873288320206676, CurrSamplesPerSec=5.354985363510395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2693/5000 [9:08:18<7:02:32, 10.99s/it][2022-12-20 19:30:48,910] [INFO] [timer.py:197:stop] 0/5452, RunningAvgSamplesPerSec=5.873280674117597, CurrSamplesPerSec=5.298236638680514, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2694/5000 [9:08:29<7:03:22, 11.02s/it][2022-12-20 19:30:59,919] [INFO] [timer.py:197:stop] 0/5454, RunningAvgSamplesPerSec=5.87328285064709, CurrSamplesPerSec=5.360196056622872, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2695/5000 [9:08:40<7:02:34, 11.00s/it][2022-12-20 19:31:10,925] [INFO] [timer.py:197:stop] 0/5456, RunningAvgSamplesPerSec=5.8732855811203954, CurrSamplesPerSec=5.321309660624183, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2696/5000 [9:08:51<7:02:48, 11.01s/it][2022-12-20 19:31:21,995] [INFO] [timer.py:197:stop] 0/5458, RunningAvgSamplesPerSec=5.873276216977695, CurrSamplesPerSec=5.292462100408798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2697/5000 [9:09:02<7:03:06, 11.02s/it][2022-12-20 19:31:33,004] [INFO] [logging.py:68:log_dist] [Rank 0] step=2730, skipped=3, lr=[5.053333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:31:33,006] [INFO] [timer.py:197:stop] 0/5460, RunningAvgSamplesPerSec=5.873278132955492, CurrSamplesPerSec=5.328033844027922, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2698/5000 [9:09:13<7:02:25, 11.01s/it][2022-12-20 19:31:43,965] [INFO] [timer.py:197:stop] 0/5462, RunningAvgSamplesPerSec=5.873290029596865, CurrSamplesPerSec=5.345760412042352, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2699/5000 [9:09:24<7:02:05, 11.01s/it][2022-12-20 19:31:54,989] [INFO] [timer.py:197:stop] 0/5464, RunningAvgSamplesPerSec=5.87328938015648, CurrSamplesPerSec=5.322120976054332, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2700/5000 [9:09:35<7:02:09, 11.01s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 5.0488888888888895e-06, 'epoch': 65.84}

 54%|█████▍    | 2700/5000 [9:09:35<7:02:09, 11.01s/it][2022-12-20 19:32:06,017] [INFO] [timer.py:197:stop] 0/5466, RunningAvgSamplesPerSec=5.873288041484228, CurrSamplesPerSec=5.32390420697494, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2701/5000 [9:09:46<7:02:05, 11.02s/it][2022-12-20 19:32:17,026] [INFO] [timer.py:197:stop] 0/5468, RunningAvgSamplesPerSec=5.873290530472754, CurrSamplesPerSec=5.335470419797562, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2702/5000 [9:09:57<7:01:49, 11.01s/it][2022-12-20 19:32:28,003] [INFO] [timer.py:197:stop] 0/5470, RunningAvgSamplesPerSec=5.873298845353902, CurrSamplesPerSec=5.363090960433488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2703/5000 [9:10:08<7:01:16, 11.00s/it][2022-12-20 19:32:39,028] [INFO] [timer.py:197:stop] 0/5472, RunningAvgSamplesPerSec=5.873297875966904, CurrSamplesPerSec=5.32386007094475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2704/5000 [9:10:19<7:01:21, 11.01s/it][2022-12-20 19:32:50,057] [INFO] [timer.py:197:stop] 0/5474, RunningAvgSamplesPerSec=5.873297006721445, CurrSamplesPerSec=5.3260383916964775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2705/5000 [9:10:30<7:01:19, 11.01s/it][2022-12-20 19:33:01,104] [INFO] [timer.py:197:stop] 0/5476, RunningAvgSamplesPerSec=5.8732918151526246, CurrSamplesPerSec=5.302880434971567, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2706/5000 [9:10:41<7:01:03, 11.01s/it][2022-12-20 19:33:11,167] [INFO] [timer.py:197:stop] 0/5478, RunningAvgSamplesPerSec=5.873478404380549, CurrSamplesPerSec=6.275279408249182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 19:33:22,199] [INFO] [logging.py:68:log_dist] [Rank 0] step=2740, skipped=3, lr=[5.031111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:33:22,201] [INFO] [timer.py:197:stop] 0/5480, RunningAvgSamplesPerSec=5.87347490645655, CurrSamplesPerSec=5.310503663229394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2707/5000 [9:10:57<8:00:12, 12.57s/it][2022-12-20 19:33:33,236] [INFO] [timer.py:197:stop] 0/5482, RunningAvgSamplesPerSec=5.873471734203726, CurrSamplesPerSec=5.320749374944079, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2708/5000 [9:11:08<7:42:27, 12.11s/it][2022-12-20 19:33:44,209] [INFO] [timer.py:197:stop] 0/5484, RunningAvgSamplesPerSec=5.873480807556759, CurrSamplesPerSec=5.371699437642947, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2709/5000 [9:11:19<7:29:16, 11.77s/it][2022-12-20 19:33:55,189] [INFO] [timer.py:197:stop] 0/5486, RunningAvgSamplesPerSec=5.8734888259086375, CurrSamplesPerSec=5.3289934110657455, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2710/5000 [9:11:30<7:20:04, 11.53s/it][2022-12-20 19:34:06,207] [INFO] [timer.py:197:stop] 0/5488, RunningAvgSamplesPerSec=5.873489169950928, CurrSamplesPerSec=5.325164609964567, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2711/5000 [9:11:41<7:14:01, 11.38s/it][2022-12-20 19:34:17,236] [INFO] [timer.py:197:stop] 0/5490, RunningAvgSamplesPerSec=5.873487250218843, CurrSamplesPerSec=5.33543584812903, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2712/5000 [9:11:52<7:09:51, 11.27s/it][2022-12-20 19:34:28,231] [INFO] [timer.py:197:stop] 0/5492, RunningAvgSamplesPerSec=5.873492282770016, CurrSamplesPerSec=5.319760724828281, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2713/5000 [9:12:03<7:06:29, 11.19s/it][2022-12-20 19:34:39,234] [INFO] [timer.py:197:stop] 0/5494, RunningAvgSamplesPerSec=5.873495409961955, CurrSamplesPerSec=5.341343494394558, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2714/5000 [9:12:14<7:04:10, 11.13s/it][2022-12-20 19:34:50,210] [INFO] [timer.py:197:stop] 0/5496, RunningAvgSamplesPerSec=5.873504014481066, CurrSamplesPerSec=5.3942010352523955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2715/5000 [9:12:25<7:02:11, 11.09s/it][2022-12-20 19:35:01,213] [INFO] [timer.py:197:stop] 0/5498, RunningAvgSamplesPerSec=5.873507213848976, CurrSamplesPerSec=5.323364699489747, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2716/5000 [9:12:36<7:01:03, 11.06s/it][2022-12-20 19:35:12,211] [INFO] [logging.py:68:log_dist] [Rank 0] step=2750, skipped=3, lr=[5.00888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:35:12,213] [INFO] [timer.py:197:stop] 0/5500, RunningAvgSamplesPerSec=5.873511101389101, CurrSamplesPerSec=5.32293718316199, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2717/5000 [9:12:47<7:00:10, 11.04s/it][2022-12-20 19:35:23,217] [INFO] [timer.py:197:stop] 0/5502, RunningAvgSamplesPerSec=5.873514018967947, CurrSamplesPerSec=5.358672112785867, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2718/5000 [9:12:58<6:59:33, 11.03s/it][2022-12-20 19:35:34,268] [INFO] [timer.py:197:stop] 0/5504, RunningAvgSamplesPerSec=5.87350800158869, CurrSamplesPerSec=5.313978962748543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2719/5000 [9:13:09<6:59:35, 11.04s/it][2022-12-20 19:35:45,257] [INFO] [timer.py:197:stop] 0/5506, RunningAvgSamplesPerSec=5.873513930115178, CurrSamplesPerSec=5.360979659526894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2720/5000 [9:13:20<6:58:52, 11.02s/it][2022-12-20 19:35:56,259] [INFO] [timer.py:197:stop] 0/5508, RunningAvgSamplesPerSec=5.873517267736895, CurrSamplesPerSec=5.338539962961313, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2721/5000 [9:13:31<6:58:26, 11.02s/it][2022-12-20 19:36:07,261] [INFO] [timer.py:197:stop] 0/5510, RunningAvgSamplesPerSec=5.873520697482273, CurrSamplesPerSec=5.341154105967314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2722/5000 [9:13:42<6:58:05, 11.01s/it][2022-12-20 19:36:18,197] [INFO] [timer.py:197:stop] 0/5512, RunningAvgSamplesPerSec=5.873536885184952, CurrSamplesPerSec=5.3780313996281155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2723/5000 [9:13:53<6:57:02, 10.99s/it][2022-12-20 19:36:29,187] [INFO] [timer.py:197:stop] 0/5514, RunningAvgSamplesPerSec=5.8735426417794665, CurrSamplesPerSec=5.36003765108807, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 54%|█████▍    | 2724/5000 [9:14:04<6:56:52, 10.99s/it][2022-12-20 19:36:40,204] [INFO] [timer.py:197:stop] 0/5516, RunningAvgSamplesPerSec=5.873543391599133, CurrSamplesPerSec=5.30801200274857, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2725/5000 [9:14:15<6:56:59, 11.00s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.991111111111112e-06, 'epoch': 66.46}

 55%|█████▍    | 2725/5000 [9:14:15<6:56:59, 11.00s/it][2022-12-20 19:36:51,205] [INFO] [timer.py:197:stop] 0/5518, RunningAvgSamplesPerSec=5.873547739027401, CurrSamplesPerSec=5.347540133725031, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2726/5000 [9:14:26<6:56:50, 11.00s/it][2022-12-20 19:37:02,176] [INFO] [logging.py:68:log_dist] [Rank 0] step=2760, skipped=3, lr=[4.986666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:37:02,177] [INFO] [timer.py:197:stop] 0/5520, RunningAvgSamplesPerSec=5.873556916619805, CurrSamplesPerSec=5.35864750909103, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2727/5000 [9:14:37<6:56:22, 10.99s/it][2022-12-20 19:37:13,201] [INFO] [timer.py:197:stop] 0/5522, RunningAvgSamplesPerSec=5.873556369180423, CurrSamplesPerSec=5.342890137698586, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2728/5000 [9:14:48<6:56:33, 11.00s/it][2022-12-20 19:37:24,195] [INFO] [timer.py:197:stop] 0/5524, RunningAvgSamplesPerSec=5.873561513267471, CurrSamplesPerSec=5.3282817413780075, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2729/5000 [9:14:59<6:56:18, 11.00s/it][2022-12-20 19:37:35,216] [INFO] [timer.py:197:stop] 0/5526, RunningAvgSamplesPerSec=5.873561310421968, CurrSamplesPerSec=5.326268771278203, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2730/5000 [9:15:10<6:56:21, 11.01s/it][2022-12-20 19:37:46,285] [INFO] [timer.py:197:stop] 0/5528, RunningAvgSamplesPerSec=5.873551749990324, CurrSamplesPerSec=5.288224459023446, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2731/5000 [9:15:21<6:56:54, 11.02s/it][2022-12-20 19:37:57,350] [INFO] [timer.py:197:stop] 0/5530, RunningAvgSamplesPerSec=5.873542956496493, CurrSamplesPerSec=5.294936255731051, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2732/5000 [9:15:32<6:57:11, 11.04s/it][2022-12-20 19:38:08,326] [INFO] [timer.py:197:stop] 0/5532, RunningAvgSamplesPerSec=5.873551340368789, CurrSamplesPerSec=5.360058414523632, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2733/5000 [9:15:43<6:56:19, 11.02s/it][2022-12-20 19:38:19,330] [INFO] [timer.py:197:stop] 0/5534, RunningAvgSamplesPerSec=5.873554694401584, CurrSamplesPerSec=5.342884395133809, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2734/5000 [9:15:54<6:55:57, 11.01s/it][2022-12-20 19:38:30,356] [INFO] [timer.py:197:stop] 0/5536, RunningAvgSamplesPerSec=5.873553591374784, CurrSamplesPerSec=5.322258576051651, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2735/5000 [9:16:05<6:55:55, 11.02s/it][2022-12-20 19:38:41,339] [INFO] [timer.py:197:stop] 0/5538, RunningAvgSamplesPerSec=5.873560905327284, CurrSamplesPerSec=5.321241306202499, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2736/5000 [9:16:16<6:55:20, 11.01s/it][2022-12-20 19:38:52,328] [INFO] [logging.py:68:log_dist] [Rank 0] step=2770, skipped=3, lr=[4.964444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:38:52,329] [INFO] [timer.py:197:stop] 0/5540, RunningAvgSamplesPerSec=5.873566896299324, CurrSamplesPerSec=5.314301092839466, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2737/5000 [9:16:27<6:54:57, 11.00s/it][2022-12-20 19:39:03,302] [INFO] [timer.py:197:stop] 0/5542, RunningAvgSamplesPerSec=5.8735761746976, CurrSamplesPerSec=5.361571795625663, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2738/5000 [9:16:38<6:54:27, 10.99s/it][2022-12-20 19:39:14,345] [INFO] [timer.py:197:stop] 0/5544, RunningAvgSamplesPerSec=5.873572219641492, CurrSamplesPerSec=5.3165741558389445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2739/5000 [9:16:49<6:54:49, 11.01s/it][2022-12-20 19:39:25,320] [INFO] [timer.py:197:stop] 0/5546, RunningAvgSamplesPerSec=5.873581127670186, CurrSamplesPerSec=5.361694950665539, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2740/5000 [9:17:00<6:54:16, 11.00s/it][2022-12-20 19:39:36,334] [INFO] [timer.py:197:stop] 0/5548, RunningAvgSamplesPerSec=5.873582498875161, CurrSamplesPerSec=5.332232172862595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2741/5000 [9:17:11<6:54:15, 11.00s/it][2022-12-20 19:39:47,396] [INFO] [timer.py:197:stop] 0/5550, RunningAvgSamplesPerSec=5.873574789539594, CurrSamplesPerSec=5.286789887746182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2742/5000 [9:17:22<6:54:44, 11.02s/it][2022-12-20 19:39:58,400] [INFO] [timer.py:197:stop] 0/5552, RunningAvgSamplesPerSec=5.873578084719662, CurrSamplesPerSec=5.354993695946194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2743/5000 [9:17:33<6:54:22, 11.02s/it][2022-12-20 19:40:09,365] [INFO] [timer.py:197:stop] 0/5554, RunningAvgSamplesPerSec=5.873589161204107, CurrSamplesPerSec=5.336683893422925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2744/5000 [9:17:44<6:53:36, 11.00s/it][2022-12-20 19:40:20,409] [INFO] [timer.py:197:stop] 0/5556, RunningAvgSamplesPerSec=5.873584460360476, CurrSamplesPerSec=5.3223045849999355, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2745/5000 [9:17:56<6:53:55, 11.01s/it][2022-12-20 19:40:31,433] [INFO] [timer.py:197:stop] 0/5558, RunningAvgSamplesPerSec=5.873584007540353, CurrSamplesPerSec=5.334101681653499, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2746/5000 [9:18:07<6:53:51, 11.02s/it][2022-12-20 19:40:42,391] [INFO] [logging.py:68:log_dist] [Rank 0] step=2780, skipped=3, lr=[4.942222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:40:42,393] [INFO] [timer.py:197:stop] 0/5560, RunningAvgSamplesPerSec=5.873595856104104, CurrSamplesPerSec=5.374349177000311, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2747/5000 [9:18:17<6:53:02, 11.00s/it][2022-12-20 19:40:52,434] [INFO] [timer.py:197:stop] 0/5562, RunningAvgSamplesPerSec=5.873784061792545, CurrSamplesPerSec=5.390280375228017, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2748/5000 [9:18:32<7:37:54, 12.20s/it][2022-12-20 19:41:03,464] [INFO] [timer.py:197:stop] 0/5564, RunningAvgSamplesPerSec=5.873782154540114, CurrSamplesPerSec=5.328951517988772, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▍    | 2749/5000 [9:18:43<7:23:53, 11.83s/it][2022-12-20 19:41:14,411] [INFO] [timer.py:197:stop] 0/5566, RunningAvgSamplesPerSec=5.873796367150424, CurrSamplesPerSec=5.351536680975152, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2750/5000 [9:18:54<7:14:24, 11.58s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.935555555555556e-06, 'epoch': 67.07}

 55%|█████▌    | 2750/5000 [9:18:54<7:14:24, 11.58s/it][2022-12-20 19:41:25,456] [INFO] [timer.py:197:stop] 0/5568, RunningAvgSamplesPerSec=5.873791776318774, CurrSamplesPerSec=5.318304780080689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2751/5000 [9:19:06<7:08:14, 11.42s/it][2022-12-20 19:41:36,508] [INFO] [timer.py:197:stop] 0/5570, RunningAvgSamplesPerSec=5.87378551825661, CurrSamplesPerSec=5.31663207086998, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2752/5000 [9:19:17<7:03:21, 11.30s/it][2022-12-20 19:41:47,473] [INFO] [timer.py:197:stop] 0/5572, RunningAvgSamplesPerSec=5.873796096979941, CurrSamplesPerSec=5.354570483729123, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2753/5000 [9:19:28<6:59:44, 11.21s/it][2022-12-20 19:41:58,533] [INFO] [timer.py:197:stop] 0/5574, RunningAvgSamplesPerSec=5.873788494292831, CurrSamplesPerSec=5.297263653758482, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2754/5000 [9:19:39<6:57:50, 11.16s/it][2022-12-20 19:42:09,535] [INFO] [timer.py:197:stop] 0/5576, RunningAvgSamplesPerSec=5.8737919261518385, CurrSamplesPerSec=5.344961881955423, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2755/5000 [9:19:50<6:55:52, 11.11s/it][2022-12-20 19:42:20,556] [INFO] [timer.py:197:stop] 0/5578, RunningAvgSamplesPerSec=5.873791728788294, CurrSamplesPerSec=5.328777816853849, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2756/5000 [9:20:01<6:54:50, 11.09s/it][2022-12-20 19:42:31,607] [INFO] [logging.py:68:log_dist] [Rank 0] step=2790, skipped=3, lr=[4.92e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:42:31,609] [INFO] [timer.py:197:stop] 0/5580, RunningAvgSamplesPerSec=5.873785311314344, CurrSamplesPerSec=5.316008129170332, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2757/5000 [9:20:12<6:53:43, 11.07s/it][2022-12-20 19:42:42,576] [INFO] [timer.py:197:stop] 0/5582, RunningAvgSamplesPerSec=5.873795462857526, CurrSamplesPerSec=5.354707416858103, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2758/5000 [9:20:23<6:52:25, 11.04s/it][2022-12-20 19:42:53,554] [INFO] [timer.py:197:stop] 0/5584, RunningAvgSamplesPerSec=5.873803465972992, CurrSamplesPerSec=5.344407457959762, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2759/5000 [9:20:34<6:51:45, 11.02s/it][2022-12-20 19:43:04,578] [INFO] [timer.py:197:stop] 0/5586, RunningAvgSamplesPerSec=5.873802423431712, CurrSamplesPerSec=5.317430373841255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2760/5000 [9:20:45<6:51:47, 11.03s/it][2022-12-20 19:43:15,608] [INFO] [timer.py:197:stop] 0/5588, RunningAvgSamplesPerSec=5.87380048815207, CurrSamplesPerSec=5.330129006634499, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2761/5000 [9:20:56<6:51:14, 11.02s/it][2022-12-20 19:43:26,648] [INFO] [timer.py:197:stop] 0/5590, RunningAvgSamplesPerSec=5.8737966535704285, CurrSamplesPerSec=5.293396370329786, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2762/5000 [9:21:07<6:51:39, 11.04s/it][2022-12-20 19:43:37,708] [INFO] [timer.py:197:stop] 0/5592, RunningAvgSamplesPerSec=5.8737886375716055, CurrSamplesPerSec=5.304641778456514, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2763/5000 [9:21:18<6:51:34, 11.04s/it][2022-12-20 19:43:48,700] [INFO] [timer.py:197:stop] 0/5594, RunningAvgSamplesPerSec=5.873793766613712, CurrSamplesPerSec=5.351837772532358, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2764/5000 [9:21:29<6:50:29, 11.01s/it][2022-12-20 19:43:59,682] [INFO] [timer.py:197:stop] 0/5596, RunningAvgSamplesPerSec=5.873801199347313, CurrSamplesPerSec=5.331814245059606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2765/5000 [9:21:40<6:50:03, 11.01s/it][2022-12-20 19:44:10,643] [INFO] [timer.py:197:stop] 0/5598, RunningAvgSamplesPerSec=5.873812509289567, CurrSamplesPerSec=5.359181567010169, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2766/5000 [9:21:51<6:49:19, 10.99s/it][2022-12-20 19:44:21,657] [INFO] [logging.py:68:log_dist] [Rank 0] step=2800, skipped=3, lr=[4.897777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:44:21,659] [INFO] [timer.py:197:stop] 0/5600, RunningAvgSamplesPerSec=5.873813294912358, CurrSamplesPerSec=5.310260568792263, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2767/5000 [9:22:02<6:49:44, 11.01s/it][2022-12-20 19:44:32,627] [INFO] [timer.py:197:stop] 0/5602, RunningAvgSamplesPerSec=5.873823350634759, CurrSamplesPerSec=5.379686477851786, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2768/5000 [9:22:13<6:49:02, 11.00s/it][2022-12-20 19:44:43,652] [INFO] [timer.py:197:stop] 0/5604, RunningAvgSamplesPerSec=5.873822587688678, CurrSamplesPerSec=5.324985662723378, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2769/5000 [9:22:24<6:48:59, 11.00s/it][2022-12-20 19:44:54,665] [INFO] [timer.py:197:stop] 0/5606, RunningAvgSamplesPerSec=5.87382371538216, CurrSamplesPerSec=5.320041592315165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2770/5000 [9:22:35<6:49:27, 11.02s/it][2022-12-20 19:45:05,733] [INFO] [timer.py:197:stop] 0/5608, RunningAvgSamplesPerSec=5.873814666235967, CurrSamplesPerSec=5.311608689198078, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2771/5000 [9:22:46<6:49:46, 11.03s/it][2022-12-20 19:45:16,836] [INFO] [timer.py:197:stop] 0/5610, RunningAvgSamplesPerSec=5.873798613325374, CurrSamplesPerSec=5.275306650567709, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2772/5000 [9:22:57<6:50:07, 11.04s/it][2022-12-20 19:45:27,857] [INFO] [timer.py:197:stop] 0/5612, RunningAvgSamplesPerSec=5.8737985908181045, CurrSamplesPerSec=5.325901652874853, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2773/5000 [9:23:08<6:49:50, 11.04s/it][2022-12-20 19:45:38,911] [INFO] [timer.py:197:stop] 0/5614, RunningAvgSamplesPerSec=5.873791739166662, CurrSamplesPerSec=5.3083934548797505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 55%|█████▌    | 2774/5000 [9:23:19<6:49:26, 11.04s/it][2022-12-20 19:45:49,891] [INFO] [timer.py:197:stop] 0/5616, RunningAvgSamplesPerSec=5.873799138445663, CurrSamplesPerSec=5.34574167546553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2775/5000 [9:23:30<6:48:50, 11.02s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.880000000000001e-06, 'epoch': 67.67}

 56%|█████▌    | 2775/5000 [9:23:30<6:48:50, 11.02s/it][2022-12-20 19:46:00,921] [INFO] [timer.py:197:stop] 0/5618, RunningAvgSamplesPerSec=5.873797035386725, CurrSamplesPerSec=5.318684762336322, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2776/5000 [9:23:41<6:48:36, 11.02s/it][2022-12-20 19:46:11,915] [INFO] [logging.py:68:log_dist] [Rank 0] step=2810, skipped=3, lr=[4.875555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:46:11,917] [INFO] [timer.py:197:stop] 0/5620, RunningAvgSamplesPerSec=5.873801293978187, CurrSamplesPerSec=5.340453634293534, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2777/5000 [9:23:52<6:48:24, 11.02s/it][2022-12-20 19:46:22,960] [INFO] [timer.py:197:stop] 0/5622, RunningAvgSamplesPerSec=5.873796478560907, CurrSamplesPerSec=5.321919864931496, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2778/5000 [9:24:03<6:48:13, 11.02s/it][2022-12-20 19:46:33,973] [INFO] [timer.py:197:stop] 0/5624, RunningAvgSamplesPerSec=5.8737972428438425, CurrSamplesPerSec=5.329389735235788, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2779/5000 [9:24:14<6:47:27, 11.01s/it][2022-12-20 19:46:44,939] [INFO] [timer.py:197:stop] 0/5626, RunningAvgSamplesPerSec=5.87380723818, CurrSamplesPerSec=5.334365620635847, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2780/5000 [9:24:25<6:46:43, 10.99s/it][2022-12-20 19:46:55,862] [INFO] [timer.py:197:stop] 0/5628, RunningAvgSamplesPerSec=5.873825495623536, CurrSamplesPerSec=5.36617585570204, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2781/5000 [9:24:36<6:46:21, 10.99s/it][2022-12-20 19:47:06,848] [INFO] [timer.py:197:stop] 0/5630, RunningAvgSamplesPerSec=5.873831663148455, CurrSamplesPerSec=5.357923191980688, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2782/5000 [9:24:47<6:46:04, 10.98s/it][2022-12-20 19:47:17,798] [INFO] [timer.py:197:stop] 0/5632, RunningAvgSamplesPerSec=5.873844738981575, CurrSamplesPerSec=5.382513113129933, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2783/5000 [9:24:58<6:45:36, 10.98s/it][2022-12-20 19:47:28,810] [INFO] [timer.py:197:stop] 0/5634, RunningAvgSamplesPerSec=5.8738460673213515, CurrSamplesPerSec=5.336412511113242, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2784/5000 [9:25:09<6:45:30, 10.98s/it][2022-12-20 19:47:39,800] [INFO] [timer.py:197:stop] 0/5636, RunningAvgSamplesPerSec=5.873851531051227, CurrSamplesPerSec=5.3320199175092124, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2785/5000 [9:25:20<6:45:47, 10.99s/it][2022-12-20 19:47:50,826] [INFO] [timer.py:197:stop] 0/5638, RunningAvgSamplesPerSec=5.8738499112889535, CurrSamplesPerSec=5.326702953576216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2786/5000 [9:25:31<6:45:49, 11.00s/it][2022-12-20 19:48:01,833] [INFO] [logging.py:68:log_dist] [Rank 0] step=2820, skipped=3, lr=[4.853333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:48:01,834] [INFO] [timer.py:197:stop] 0/5640, RunningAvgSamplesPerSec=5.873851931605211, CurrSamplesPerSec=5.330478078345613, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2787/5000 [9:25:42<6:45:31, 10.99s/it][2022-12-20 19:48:12,806] [INFO] [timer.py:197:stop] 0/5642, RunningAvgSamplesPerSec=5.873860829070926, CurrSamplesPerSec=5.344046771672473, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2788/5000 [9:25:53<6:45:26, 11.00s/it][2022-12-20 19:48:22,907] [INFO] [timer.py:197:stop] 0/5644, RunningAvgSamplesPerSec=5.874034392041914, CurrSamplesPerSec=6.280781027546057, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 19:48:33,912] [INFO] [timer.py:197:stop] 0/5646, RunningAvgSamplesPerSec=5.874037323258759, CurrSamplesPerSec=5.3153839128937115, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2789/5000 [9:26:09<7:42:15, 12.54s/it][2022-12-20 19:48:44,878] [INFO] [timer.py:197:stop] 0/5648, RunningAvgSamplesPerSec=5.874047515144264, CurrSamplesPerSec=5.363802313643479, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2790/5000 [9:26:20<7:24:36, 12.07s/it][2022-12-20 19:48:55,930] [INFO] [timer.py:197:stop] 0/5650, RunningAvgSamplesPerSec=5.874041232593043, CurrSamplesPerSec=5.3037232356404855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2791/5000 [9:26:31<7:13:09, 11.77s/it][2022-12-20 19:49:06,909] [INFO] [timer.py:197:stop] 0/5652, RunningAvgSamplesPerSec=5.874048922961252, CurrSamplesPerSec=5.357295936736377, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2792/5000 [9:26:42<7:04:16, 11.53s/it][2022-12-20 19:49:17,841] [INFO] [timer.py:197:stop] 0/5654, RunningAvgSamplesPerSec=5.874065682504712, CurrSamplesPerSec=5.365903825480039, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2793/5000 [9:26:53<6:57:30, 11.35s/it][2022-12-20 19:49:28,803] [INFO] [timer.py:197:stop] 0/5656, RunningAvgSamplesPerSec=5.874076387249856, CurrSamplesPerSec=5.314600744650652, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2794/5000 [9:27:04<6:53:01, 11.23s/it][2022-12-20 19:49:39,850] [INFO] [timer.py:197:stop] 0/5658, RunningAvgSamplesPerSec=5.874071170541584, CurrSamplesPerSec=5.266305597013842, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2795/5000 [9:27:15<6:50:46, 11.18s/it][2022-12-20 19:49:50,921] [INFO] [logging.py:68:log_dist] [Rank 0] step=2830, skipped=3, lr=[4.831111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:49:50,923] [INFO] [timer.py:197:stop] 0/5660, RunningAvgSamplesPerSec=5.87406074746877, CurrSamplesPerSec=5.293036274159943, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2796/5000 [9:27:26<6:49:26, 11.15s/it][2022-12-20 19:50:01,950] [INFO] [timer.py:197:stop] 0/5662, RunningAvgSamplesPerSec=5.8740592631947175, CurrSamplesPerSec=5.322980037257525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2797/5000 [9:27:37<6:47:56, 11.11s/it][2022-12-20 19:50:12,988] [INFO] [timer.py:197:stop] 0/5664, RunningAvgSamplesPerSec=5.874055553027704, CurrSamplesPerSec=5.313397081453832, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2798/5000 [9:27:48<6:46:57, 11.09s/it][2022-12-20 19:50:23,957] [INFO] [timer.py:197:stop] 0/5666, RunningAvgSamplesPerSec=5.8740650716204605, CurrSamplesPerSec=5.335225670509933, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2799/5000 [9:27:59<6:45:27, 11.05s/it][2022-12-20 19:50:34,903] [INFO] [timer.py:197:stop] 0/5668, RunningAvgSamplesPerSec=5.874079108070821, CurrSamplesPerSec=5.353138552953794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2800/5000 [9:28:10<6:44:05, 11.02s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.822222222222222e-06, 'epoch': 68.29}

 56%|█████▌    | 2800/5000 [9:28:10<6:44:05, 11.02s/it][2022-12-20 19:50:45,907] [INFO] [timer.py:197:stop] 0/5670, RunningAvgSamplesPerSec=5.874081955942947, CurrSamplesPerSec=5.340098580636496, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2801/5000 [9:28:21<6:43:43, 11.02s/it][2022-12-20 19:50:56,877] [INFO] [timer.py:197:stop] 0/5672, RunningAvgSamplesPerSec=5.874091433743807, CurrSamplesPerSec=5.371057131335535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2802/5000 [9:28:32<6:43:02, 11.00s/it][2022-12-20 19:51:07,908] [INFO] [timer.py:197:stop] 0/5674, RunningAvgSamplesPerSec=5.874088896813064, CurrSamplesPerSec=5.314274580331696, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2803/5000 [9:28:43<6:43:10, 11.01s/it][2022-12-20 19:51:18,909] [INFO] [timer.py:197:stop] 0/5676, RunningAvgSamplesPerSec=5.874092199611981, CurrSamplesPerSec=5.31110361552119, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2804/5000 [9:28:54<6:42:53, 11.01s/it][2022-12-20 19:51:29,935] [INFO] [timer.py:197:stop] 0/5678, RunningAvgSamplesPerSec=5.874090597320321, CurrSamplesPerSec=5.315743056188142, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2805/5000 [9:29:05<6:42:54, 11.01s/it][2022-12-20 19:51:40,985] [INFO] [logging.py:68:log_dist] [Rank 0] step=2840, skipped=3, lr=[4.808888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:51:40,987] [INFO] [timer.py:197:stop] 0/5680, RunningAvgSamplesPerSec=5.874084296384431, CurrSamplesPerSec=5.313309789216538, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2806/5000 [9:29:16<6:43:08, 11.02s/it][2022-12-20 19:51:51,990] [INFO] [timer.py:197:stop] 0/5682, RunningAvgSamplesPerSec=5.874087462115047, CurrSamplesPerSec=5.324057949674623, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2807/5000 [9:29:27<6:42:42, 11.02s/it][2022-12-20 19:52:02,981] [INFO] [timer.py:197:stop] 0/5684, RunningAvgSamplesPerSec=5.874092943791963, CurrSamplesPerSec=5.333381229406907, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2808/5000 [9:29:38<6:42:13, 11.01s/it][2022-12-20 19:52:13,994] [INFO] [timer.py:197:stop] 0/5686, RunningAvgSamplesPerSec=5.87409428822226, CurrSamplesPerSec=5.344357661203878, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2809/5000 [9:29:49<6:42:04, 11.01s/it][2022-12-20 19:52:25,039] [INFO] [timer.py:197:stop] 0/5688, RunningAvgSamplesPerSec=5.874089613519568, CurrSamplesPerSec=5.325526766680763, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2810/5000 [9:30:00<6:42:16, 11.02s/it][2022-12-20 19:52:36,059] [INFO] [timer.py:197:stop] 0/5690, RunningAvgSamplesPerSec=5.874089342463575, CurrSamplesPerSec=5.351310084443843, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2811/5000 [9:30:11<6:42:04, 11.02s/it][2022-12-20 19:52:47,093] [INFO] [timer.py:197:stop] 0/5692, RunningAvgSamplesPerSec=5.874086672425911, CurrSamplesPerSec=5.314015360860494, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▌    | 2812/5000 [9:30:22<6:42:02, 11.02s/it][2022-12-20 19:52:58,160] [INFO] [timer.py:197:stop] 0/5694, RunningAvgSamplesPerSec=5.874077738030854, CurrSamplesPerSec=5.307387773955233, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2813/5000 [9:30:33<6:42:19, 11.04s/it][2022-12-20 19:53:09,186] [INFO] [timer.py:197:stop] 0/5696, RunningAvgSamplesPerSec=5.874076500856282, CurrSamplesPerSec=5.321241306202499, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2814/5000 [9:30:44<6:42:00, 11.03s/it][2022-12-20 19:53:20,201] [INFO] [timer.py:197:stop] 0/5698, RunningAvgSamplesPerSec=5.874077279176397, CurrSamplesPerSec=5.316346719924513, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2815/5000 [9:30:55<6:41:36, 11.03s/it][2022-12-20 19:53:31,221] [INFO] [logging.py:68:log_dist] [Rank 0] step=2850, skipped=3, lr=[4.786666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:53:31,222] [INFO] [timer.py:197:stop] 0/5700, RunningAvgSamplesPerSec=5.874076700486542, CurrSamplesPerSec=5.3059660712079655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2816/5000 [9:31:06<6:41:21, 11.03s/it][2022-12-20 19:53:42,239] [INFO] [timer.py:197:stop] 0/5702, RunningAvgSamplesPerSec=5.8740770406572125, CurrSamplesPerSec=5.299960156829668, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2817/5000 [9:31:17<6:41:04, 11.02s/it][2022-12-20 19:53:53,321] [INFO] [timer.py:197:stop] 0/5704, RunningAvgSamplesPerSec=5.874065554450859, CurrSamplesPerSec=5.306221778839163, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2818/5000 [9:31:28<6:41:31, 11.04s/it][2022-12-20 19:54:04,303] [INFO] [timer.py:197:stop] 0/5706, RunningAvgSamplesPerSec=5.874072414392515, CurrSamplesPerSec=5.3202275884592, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2819/5000 [9:31:39<6:40:41, 11.02s/it][2022-12-20 19:54:15,297] [INFO] [timer.py:197:stop] 0/5708, RunningAvgSamplesPerSec=5.874077129048862, CurrSamplesPerSec=5.357236490909987, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2820/5000 [9:31:50<6:40:11, 11.01s/it][2022-12-20 19:54:26,249] [INFO] [timer.py:197:stop] 0/5710, RunningAvgSamplesPerSec=5.874089910910046, CurrSamplesPerSec=5.362374225584863, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2821/5000 [9:32:01<6:39:19, 11.00s/it][2022-12-20 19:54:37,267] [INFO] [timer.py:197:stop] 0/5712, RunningAvgSamplesPerSec=5.874090321363082, CurrSamplesPerSec=5.351450051410196, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2822/5000 [9:32:12<6:39:22, 11.00s/it][2022-12-20 19:54:48,297] [INFO] [timer.py:197:stop] 0/5714, RunningAvgSamplesPerSec=5.8740880477430695, CurrSamplesPerSec=5.316479809690409, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2823/5000 [9:32:23<6:39:30, 11.01s/it][2022-12-20 19:54:59,365] [INFO] [timer.py:197:stop] 0/5716, RunningAvgSamplesPerSec=5.874078942640688, CurrSamplesPerSec=5.310364149093905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2824/5000 [9:32:34<6:39:56, 11.03s/it][2022-12-20 19:55:10,353] [INFO] [timer.py:197:stop] 0/5718, RunningAvgSamplesPerSec=5.874084770268875, CurrSamplesPerSec=5.359720226014547, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 56%|█████▋    | 2825/5000 [9:32:45<6:39:19, 11.02s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.766666666666667e-06, 'epoch': 68.89}

 56%|█████▋    | 2825/5000 [9:32:45<6:39:19, 11.02s/it][2022-12-20 19:55:21,306] [INFO] [logging.py:68:log_dist] [Rank 0] step=2860, skipped=3, lr=[4.7644444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:55:21,308] [INFO] [timer.py:197:stop] 0/5720, RunningAvgSamplesPerSec=5.8740969873446165, CurrSamplesPerSec=5.323027536445529, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2826/5000 [9:32:56<6:38:28, 11.00s/it][2022-12-20 19:55:32,352] [INFO] [timer.py:197:stop] 0/5722, RunningAvgSamplesPerSec=5.874092365053945, CurrSamplesPerSec=5.311949242519917, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2827/5000 [9:33:07<6:38:48, 11.01s/it][2022-12-20 19:55:43,381] [INFO] [timer.py:197:stop] 0/5724, RunningAvgSamplesPerSec=5.874090600049113, CurrSamplesPerSec=5.29813353093577, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2828/5000 [9:33:18<6:38:48, 11.02s/it][2022-12-20 19:55:54,379] [INFO] [timer.py:197:stop] 0/5726, RunningAvgSamplesPerSec=5.874094592327143, CurrSamplesPerSec=5.348733738978411, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2829/5000 [9:33:29<6:38:25, 11.01s/it][2022-12-20 19:56:04,496] [INFO] [timer.py:197:stop] 0/5728, RunningAvgSamplesPerSec=5.874262808502337, CurrSamplesPerSec=5.310570901575081, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2830/5000 [9:33:45<7:22:20, 12.23s/it][2022-12-20 19:56:15,520] [INFO] [timer.py:197:stop] 0/5730, RunningAvgSamplesPerSec=5.874261923899835, CurrSamplesPerSec=5.332497621750029, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2831/5000 [9:33:56<7:08:31, 11.85s/it][2022-12-20 19:56:26,464] [INFO] [timer.py:197:stop] 0/5732, RunningAvgSamplesPerSec=5.874275811394911, CurrSamplesPerSec=5.359821035972468, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2832/5000 [9:34:07<6:58:54, 11.59s/it][2022-12-20 19:56:37,461] [INFO] [timer.py:197:stop] 0/5734, RunningAvgSamplesPerSec=5.874279911569508, CurrSamplesPerSec=5.349697365502477, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2833/5000 [9:34:18<6:52:27, 11.42s/it][2022-12-20 19:56:48,449] [INFO] [timer.py:197:stop] 0/5736, RunningAvgSamplesPerSec=5.874285631114197, CurrSamplesPerSec=5.374792094701099, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2834/5000 [9:34:28<6:46:55, 11.27s/it][2022-12-20 19:56:59,403] [INFO] [timer.py:197:stop] 0/5738, RunningAvgSamplesPerSec=5.874297596453894, CurrSamplesPerSec=5.350599054392578, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2835/5000 [9:34:39<6:43:42, 11.19s/it][2022-12-20 19:57:10,404] [INFO] [logging.py:68:log_dist] [Rank 0] step=2870, skipped=3, lr=[4.742222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:57:10,405] [INFO] [timer.py:197:stop] 0/5740, RunningAvgSamplesPerSec=5.874300721749914, CurrSamplesPerSec=5.342114365213371, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2836/5000 [9:34:50<6:41:12, 11.12s/it][2022-12-20 19:57:21,381] [INFO] [timer.py:197:stop] 0/5742, RunningAvgSamplesPerSec=5.874308922431549, CurrSamplesPerSec=5.3413607121903, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2837/5000 [9:35:01<6:39:23, 11.08s/it][2022-12-20 19:57:32,373] [INFO] [timer.py:197:stop] 0/5744, RunningAvgSamplesPerSec=5.87431381226725, CurrSamplesPerSec=5.324671530599606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2838/5000 [9:35:12<6:38:36, 11.06s/it][2022-12-20 19:57:43,367] [INFO] [timer.py:197:stop] 0/5746, RunningAvgSamplesPerSec=5.874318580181258, CurrSamplesPerSec=5.351462640261471, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2839/5000 [9:35:23<6:37:41, 11.04s/it][2022-12-20 19:57:54,396] [INFO] [timer.py:197:stop] 0/5748, RunningAvgSamplesPerSec=5.874316732306779, CurrSamplesPerSec=5.320914326318702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2840/5000 [9:35:34<6:36:32, 11.02s/it][2022-12-20 19:58:05,308] [INFO] [timer.py:197:stop] 0/5750, RunningAvgSamplesPerSec=5.874336907249694, CurrSamplesPerSec=5.3576851470414475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2841/5000 [9:35:45<6:36:18, 11.01s/it][2022-12-20 19:58:16,316] [INFO] [timer.py:197:stop] 0/5752, RunningAvgSamplesPerSec=5.874339099919703, CurrSamplesPerSec=5.3583129212701515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2842/5000 [9:35:56<6:35:55, 11.01s/it][2022-12-20 19:58:27,337] [INFO] [timer.py:197:stop] 0/5754, RunningAvgSamplesPerSec=5.87433871573882, CurrSamplesPerSec=5.335525565717599, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2843/5000 [9:36:07<6:35:48, 11.01s/it][2022-12-20 19:58:38,328] [INFO] [timer.py:197:stop] 0/5756, RunningAvgSamplesPerSec=5.874344109108672, CurrSamplesPerSec=5.3569016516554875, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2844/5000 [9:36:18<6:35:00, 10.99s/it][2022-12-20 19:58:49,304] [INFO] [timer.py:197:stop] 0/5758, RunningAvgSamplesPerSec=5.874352030077367, CurrSamplesPerSec=5.3351187853181345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2845/5000 [9:36:29<6:34:55, 11.00s/it][2022-12-20 19:59:00,297] [INFO] [logging.py:68:log_dist] [Rank 0] step=2880, skipped=3, lr=[4.7200000000000005e-06], mom=[[0.9, 0.999]]
[2022-12-20 19:59:00,299] [INFO] [timer.py:197:stop] 0/5760, RunningAvgSamplesPerSec=5.874356041967765, CurrSamplesPerSec=5.341083752969541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2846/5000 [9:36:40<6:34:31, 10.99s/it][2022-12-20 19:59:11,270] [INFO] [timer.py:197:stop] 0/5762, RunningAvgSamplesPerSec=5.8743645345354185, CurrSamplesPerSec=5.3454372244994195, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2847/5000 [9:36:51<6:34:10, 10.98s/it][2022-12-20 19:59:22,234] [INFO] [timer.py:197:stop] 0/5764, RunningAvgSamplesPerSec=5.8743746694006695, CurrSamplesPerSec=5.354211628258398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2848/5000 [9:37:02<6:34:01, 10.99s/it][2022-12-20 19:59:33,276] [INFO] [timer.py:197:stop] 0/5766, RunningAvgSamplesPerSec=5.874369968611366, CurrSamplesPerSec=5.30543942051699, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2849/5000 [9:37:13<6:34:22, 11.00s/it][2022-12-20 19:59:44,240] [INFO] [timer.py:197:stop] 0/5768, RunningAvgSamplesPerSec=5.8743798423328535, CurrSamplesPerSec=5.3689341857210655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2850/5000 [9:37:24<6:33:37, 10.98s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.711111111111111e-06, 'epoch': 69.51}

 57%|█████▋    | 2850/5000 [9:37:24<6:33:37, 10.98s/it][2022-12-20 19:59:55,218] [INFO] [timer.py:197:stop] 0/5770, RunningAvgSamplesPerSec=5.874387325776765, CurrSamplesPerSec=5.342818037502982, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2851/5000 [9:37:35<6:33:40, 10.99s/it][2022-12-20 20:00:06,284] [INFO] [timer.py:197:stop] 0/5772, RunningAvgSamplesPerSec=5.8743785660706545, CurrSamplesPerSec=5.29163810433456, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2852/5000 [9:37:46<6:34:14, 11.01s/it][2022-12-20 20:00:17,332] [INFO] [timer.py:197:stop] 0/5774, RunningAvgSamplesPerSec=5.87437272401746, CurrSamplesPerSec=5.301045096100744, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2853/5000 [9:37:57<6:34:30, 11.02s/it][2022-12-20 20:00:28,380] [INFO] [timer.py:197:stop] 0/5776, RunningAvgSamplesPerSec=5.874367000237943, CurrSamplesPerSec=5.307387354213918, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2854/5000 [9:38:08<6:34:42, 11.04s/it][2022-12-20 20:00:39,434] [INFO] [timer.py:197:stop] 0/5778, RunningAvgSamplesPerSec=5.874360095593777, CurrSamplesPerSec=5.3126557145680495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2855/5000 [9:38:19<6:34:06, 11.02s/it][2022-12-20 20:00:50,466] [INFO] [logging.py:68:log_dist] [Rank 0] step=2890, skipped=3, lr=[4.697777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:00:50,468] [INFO] [timer.py:197:stop] 0/5780, RunningAvgSamplesPerSec=5.874356978811214, CurrSamplesPerSec=5.280061948740124, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2856/5000 [9:38:30<6:34:10, 11.03s/it][2022-12-20 20:01:01,490] [INFO] [timer.py:197:stop] 0/5782, RunningAvgSamplesPerSec=5.874355798659134, CurrSamplesPerSec=5.301830766347144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2857/5000 [9:38:42<6:34:15, 11.04s/it][2022-12-20 20:01:12,481] [INFO] [timer.py:197:stop] 0/5784, RunningAvgSamplesPerSec=5.874360896035313, CurrSamplesPerSec=5.360804934771974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2858/5000 [9:38:53<6:33:12, 11.01s/it][2022-12-20 20:01:23,486] [INFO] [timer.py:197:stop] 0/5786, RunningAvgSamplesPerSec=5.8743632233404925, CurrSamplesPerSec=5.318876565458222, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2859/5000 [9:39:04<6:33:29, 11.03s/it][2022-12-20 20:01:34,551] [INFO] [timer.py:197:stop] 0/5788, RunningAvgSamplesPerSec=5.874354271060342, CurrSamplesPerSec=5.312288157445873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2860/5000 [9:39:15<6:33:12, 11.02s/it][2022-12-20 20:01:45,564] [INFO] [timer.py:197:stop] 0/5790, RunningAvgSamplesPerSec=5.874354954865868, CurrSamplesPerSec=5.315766214807008, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2861/5000 [9:39:26<6:32:54, 11.02s/it][2022-12-20 20:01:56,574] [INFO] [timer.py:197:stop] 0/5792, RunningAvgSamplesPerSec=5.874356617019064, CurrSamplesPerSec=5.3200865085602045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2862/5000 [9:39:37<6:32:46, 11.02s/it][2022-12-20 20:02:07,617] [INFO] [timer.py:197:stop] 0/5794, RunningAvgSamplesPerSec=5.874352035993619, CurrSamplesPerSec=5.305779602675689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2863/5000 [9:39:48<6:32:24, 11.02s/it][2022-12-20 20:02:18,587] [INFO] [timer.py:197:stop] 0/5796, RunningAvgSamplesPerSec=5.874360896817245, CurrSamplesPerSec=5.3368224596949165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2864/5000 [9:39:59<6:32:07, 11.01s/it][2022-12-20 20:02:29,619] [INFO] [timer.py:197:stop] 0/5798, RunningAvgSamplesPerSec=5.874358246246645, CurrSamplesPerSec=5.315409383943446, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2865/5000 [9:40:10<6:31:54, 11.01s/it][2022-12-20 20:02:40,662] [INFO] [logging.py:68:log_dist] [Rank 0] step=2900, skipped=3, lr=[4.675555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:02:40,664] [INFO] [timer.py:197:stop] 0/5800, RunningAvgSamplesPerSec=5.874353288484792, CurrSamplesPerSec=5.286806755630848, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2866/5000 [9:40:21<6:31:51, 11.02s/it][2022-12-20 20:02:51,658] [INFO] [timer.py:197:stop] 0/5802, RunningAvgSamplesPerSec=5.874357731685904, CurrSamplesPerSec=5.315508744421285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2867/5000 [9:40:32<6:31:49, 11.02s/it][2022-12-20 20:03:02,648] [INFO] [timer.py:197:stop] 0/5804, RunningAvgSamplesPerSec=5.874362961404516, CurrSamplesPerSec=5.351406524313069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2868/5000 [9:40:43<6:31:04, 11.01s/it][2022-12-20 20:03:13,604] [INFO] [timer.py:197:stop] 0/5806, RunningAvgSamplesPerSec=5.874374559195766, CurrSamplesPerSec=5.363802527999487, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2869/5000 [9:40:54<6:30:23, 10.99s/it][2022-12-20 20:03:24,593] [INFO] [timer.py:197:stop] 0/5808, RunningAvgSamplesPerSec=5.874379684424819, CurrSamplesPerSec=5.33495931378749, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2870/5000 [9:41:05<6:30:13, 10.99s/it][2022-12-20 20:03:34,670] [INFO] [timer.py:197:stop] 0/5810, RunningAvgSamplesPerSec=5.874552452548534, CurrSamplesPerSec=6.285439165149174, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 20:03:45,726] [INFO] [timer.py:197:stop] 0/5812, RunningAvgSamplesPerSec=5.874545059490975, CurrSamplesPerSec=5.301394347192858, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2871/5000 [9:41:21<7:25:28, 12.55s/it][2022-12-20 20:03:56,774] [INFO] [timer.py:197:stop] 0/5814, RunningAvgSamplesPerSec=5.874539375564316, CurrSamplesPerSec=5.308503471073308, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2872/5000 [9:41:32<7:09:14, 12.10s/it][2022-12-20 20:04:07,759] [INFO] [timer.py:197:stop] 0/5816, RunningAvgSamplesPerSec=5.874545262649012, CurrSamplesPerSec=5.3318873194473895, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2873/5000 [9:41:43<6:57:09, 11.77s/it][2022-12-20 20:04:18,713] [INFO] [timer.py:197:stop] 0/5818, RunningAvgSamplesPerSec=5.874557316502126, CurrSamplesPerSec=5.364406436885103, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▋    | 2874/5000 [9:41:54<6:48:18, 11.52s/it][2022-12-20 20:04:29,731] [INFO] [logging.py:68:log_dist] [Rank 0] step=2910, skipped=3, lr=[4.653333333333333e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:04:29,733] [INFO] [timer.py:197:stop] 0/5820, RunningAvgSamplesPerSec=5.874556681895778, CurrSamplesPerSec=5.322935072140634, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 57%|█████▊    | 2875/5000 [9:42:05<6:42:46, 11.37s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.653333333333333e-06, 'epoch': 70.12}

 57%|█████▊    | 2875/5000 [9:42:05<6:42:46, 11.37s/it][2022-12-20 20:04:40,690] [INFO] [timer.py:197:stop] 0/5822, RunningAvgSamplesPerSec=5.874568113956616, CurrSamplesPerSec=5.360720574200327, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2876/5000 [9:42:16<6:38:10, 11.25s/it][2022-12-20 20:04:51,734] [INFO] [timer.py:197:stop] 0/5824, RunningAvgSamplesPerSec=5.874563092787816, CurrSamplesPerSec=5.32181709942815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2877/5000 [9:42:27<6:35:49, 11.19s/it][2022-12-20 20:05:02,715] [INFO] [timer.py:197:stop] 0/5826, RunningAvgSamplesPerSec=5.874569945836367, CurrSamplesPerSec=5.335487175552704, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2878/5000 [9:42:38<6:33:26, 11.12s/it][2022-12-20 20:05:13,744] [INFO] [timer.py:197:stop] 0/5828, RunningAvgSamplesPerSec=5.874567711884025, CurrSamplesPerSec=5.321493424581322, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2879/5000 [9:42:49<6:32:15, 11.10s/it][2022-12-20 20:05:24,761] [INFO] [timer.py:197:stop] 0/5830, RunningAvgSamplesPerSec=5.8745678265348635, CurrSamplesPerSec=5.336573979200925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2880/5000 [9:43:00<6:31:13, 11.07s/it][2022-12-20 20:05:35,758] [INFO] [timer.py:197:stop] 0/5832, RunningAvgSamplesPerSec=5.87457184003875, CurrSamplesPerSec=5.309804695415316, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2881/5000 [9:43:11<6:30:14, 11.05s/it][2022-12-20 20:05:46,727] [INFO] [timer.py:197:stop] 0/5834, RunningAvgSamplesPerSec=5.874580827909812, CurrSamplesPerSec=5.341647054028996, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2882/5000 [9:43:22<6:29:12, 11.03s/it][2022-12-20 20:05:57,713] [INFO] [timer.py:197:stop] 0/5836, RunningAvgSamplesPerSec=5.874586477108707, CurrSamplesPerSec=5.312242111254212, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2883/5000 [9:43:33<6:28:35, 11.01s/it][2022-12-20 20:06:08,755] [INFO] [timer.py:197:stop] 0/5838, RunningAvgSamplesPerSec=5.874582069666755, CurrSamplesPerSec=5.30614521080266, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2884/5000 [9:43:44<6:28:42, 11.02s/it][2022-12-20 20:06:19,739] [INFO] [logging.py:68:log_dist] [Rank 0] step=2920, skipped=3, lr=[4.631111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:06:19,740] [INFO] [timer.py:197:stop] 0/5840, RunningAvgSamplesPerSec=5.874587815959889, CurrSamplesPerSec=5.316077823119983, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2885/5000 [9:43:55<6:28:08, 11.01s/it][2022-12-20 20:06:30,750] [INFO] [timer.py:197:stop] 0/5842, RunningAvgSamplesPerSec=5.874589267807861, CurrSamplesPerSec=5.331193087746084, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2886/5000 [9:44:06<6:27:56, 11.01s/it][2022-12-20 20:06:41,728] [INFO] [timer.py:197:stop] 0/5844, RunningAvgSamplesPerSec=5.874596574107854, CurrSamplesPerSec=5.365968397981047, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2887/5000 [9:44:17<6:27:24, 11.00s/it][2022-12-20 20:06:52,735] [INFO] [timer.py:197:stop] 0/5846, RunningAvgSamplesPerSec=5.874598268007782, CurrSamplesPerSec=5.340028680362362, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2888/5000 [9:44:28<6:27:18, 11.00s/it][2022-12-20 20:07:03,743] [INFO] [timer.py:197:stop] 0/5848, RunningAvgSamplesPerSec=5.874600249676044, CurrSamplesPerSec=5.335969956175687, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2889/5000 [9:44:39<6:27:09, 11.00s/it][2022-12-20 20:07:14,707] [INFO] [timer.py:197:stop] 0/5850, RunningAvgSamplesPerSec=5.874609959319911, CurrSamplesPerSec=5.354375029695503, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2890/5000 [9:44:50<6:26:33, 10.99s/it][2022-12-20 20:07:25,693] [INFO] [timer.py:197:stop] 0/5852, RunningAvgSamplesPerSec=5.874615827677498, CurrSamplesPerSec=5.336775987202972, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2891/5000 [9:45:01<6:26:18, 10.99s/it][2022-12-20 20:07:36,671] [INFO] [timer.py:197:stop] 0/5854, RunningAvgSamplesPerSec=5.87462319179014, CurrSamplesPerSec=5.342575378279608, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2892/5000 [9:45:12<6:25:59, 10.99s/it][2022-12-20 20:07:47,672] [INFO] [timer.py:197:stop] 0/5856, RunningAvgSamplesPerSec=5.8746263533278364, CurrSamplesPerSec=5.3509781189051955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2893/5000 [9:45:23<6:25:58, 10.99s/it][2022-12-20 20:07:58,676] [INFO] [timer.py:197:stop] 0/5858, RunningAvgSamplesPerSec=5.874629115116678, CurrSamplesPerSec=5.308491713399877, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2894/5000 [9:45:34<6:25:55, 10.99s/it][2022-12-20 20:08:09,649] [INFO] [logging.py:68:log_dist] [Rank 0] step=2930, skipped=3, lr=[4.6088888888888895e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:08:09,650] [INFO] [timer.py:197:stop] 0/5860, RunningAvgSamplesPerSec=5.874637289752536, CurrSamplesPerSec=5.337249874807918, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2895/5000 [9:45:45<6:25:31, 10.99s/it][2022-12-20 20:08:20,574] [INFO] [timer.py:197:stop] 0/5862, RunningAvgSamplesPerSec=5.87465474384245, CurrSamplesPerSec=5.375662647147338, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2896/5000 [9:45:56<6:24:39, 10.97s/it][2022-12-20 20:08:31,598] [INFO] [timer.py:197:stop] 0/5864, RunningAvgSamplesPerSec=5.874653585468551, CurrSamplesPerSec=5.344068475311295, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2897/5000 [9:46:07<6:25:02, 10.99s/it][2022-12-20 20:08:42,624] [INFO] [timer.py:197:stop] 0/5866, RunningAvgSamplesPerSec=5.874652133087903, CurrSamplesPerSec=5.316332189957959, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2898/5000 [9:46:18<6:25:17, 11.00s/it][2022-12-20 20:08:53,681] [INFO] [timer.py:197:stop] 0/5868, RunningAvgSamplesPerSec=5.874644986601471, CurrSamplesPerSec=5.306703473929222, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2899/5000 [9:46:29<6:25:43, 11.02s/it][2022-12-20 20:09:04,706] [INFO] [timer.py:197:stop] 0/5870, RunningAvgSamplesPerSec=5.874643603436264, CurrSamplesPerSec=5.333013765622832, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2900/5000 [9:46:40<6:25:38, 11.02s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.597777777777778e-06, 'epoch': 70.72}

 58%|█████▊    | 2900/5000 [9:46:40<6:25:38, 11.02s/it][2022-12-20 20:09:15,760] [INFO] [timer.py:197:stop] 0/5872, RunningAvgSamplesPerSec=5.874637328605862, CurrSamplesPerSec=5.29676130436869, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2901/5000 [9:46:51<6:25:49, 11.03s/it][2022-12-20 20:09:26,755] [INFO] [timer.py:197:stop] 0/5874, RunningAvgSamplesPerSec=5.87464149795436, CurrSamplesPerSec=5.330367149449948, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2902/5000 [9:47:02<6:25:17, 11.02s/it][2022-12-20 20:09:37,817] [INFO] [timer.py:197:stop] 0/5876, RunningAvgSamplesPerSec=5.874633583260283, CurrSamplesPerSec=5.286907756945132, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2903/5000 [9:47:13<6:25:33, 11.03s/it][2022-12-20 20:09:48,862] [INFO] [timer.py:197:stop] 0/5878, RunningAvgSamplesPerSec=5.874629010129903, CurrSamplesPerSec=5.31441198532244, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2904/5000 [9:47:24<6:25:30, 11.04s/it][2022-12-20 20:09:59,894] [INFO] [logging.py:68:log_dist] [Rank 0] step=2940, skipped=3, lr=[4.586666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:09:59,895] [INFO] [timer.py:197:stop] 0/5880, RunningAvgSamplesPerSec=5.874626239363037, CurrSamplesPerSec=5.324101033012881, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2905/5000 [9:47:35<6:25:18, 11.04s/it][2022-12-20 20:10:10,904] [INFO] [timer.py:197:stop] 0/5882, RunningAvgSamplesPerSec=5.874628065696858, CurrSamplesPerSec=5.338482631341345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2906/5000 [9:47:46<6:24:51, 11.03s/it][2022-12-20 20:10:21,951] [INFO] [timer.py:197:stop] 0/5884, RunningAvgSamplesPerSec=5.874622932751447, CurrSamplesPerSec=5.32160419543948, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2907/5000 [9:47:57<6:24:52, 11.03s/it][2022-12-20 20:10:32,962] [INFO] [timer.py:197:stop] 0/5886, RunningAvgSamplesPerSec=5.874624177323628, CurrSamplesPerSec=5.333100858861519, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2908/5000 [9:48:08<6:24:27, 11.03s/it][2022-12-20 20:10:44,036] [INFO] [timer.py:197:stop] 0/5888, RunningAvgSamplesPerSec=5.874614211948825, CurrSamplesPerSec=5.298494530184896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2909/5000 [9:48:19<6:24:46, 11.04s/it][2022-12-20 20:10:55,048] [INFO] [timer.py:197:stop] 0/5890, RunningAvgSamplesPerSec=5.874615239935691, CurrSamplesPerSec=5.301244632606311, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2910/5000 [9:48:30<6:24:17, 11.03s/it][2022-12-20 20:11:06,001] [INFO] [timer.py:197:stop] 0/5892, RunningAvgSamplesPerSec=5.874627084469442, CurrSamplesPerSec=5.355306928879785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2911/5000 [9:48:41<6:23:16, 11.01s/it][2022-12-20 20:11:16,106] [INFO] [timer.py:197:stop] 0/5894, RunningAvgSamplesPerSec=5.874792395484535, CurrSamplesPerSec=5.3433325649823855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2912/5000 [9:48:56<7:05:18, 12.22s/it][2022-12-20 20:11:27,167] [INFO] [timer.py:197:stop] 0/5896, RunningAvgSamplesPerSec=5.874784515501794, CurrSamplesPerSec=5.289426540318353, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2913/5000 [9:49:07<6:53:07, 11.88s/it][2022-12-20 20:11:38,205] [INFO] [timer.py:197:stop] 0/5898, RunningAvgSamplesPerSec=5.874781057310203, CurrSamplesPerSec=5.321837567805312, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2914/5000 [9:49:18<6:43:54, 11.62s/it][2022-12-20 20:11:49,180] [INFO] [logging.py:68:log_dist] [Rank 0] step=2950, skipped=3, lr=[4.564444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:11:49,183] [INFO] [timer.py:197:stop] 0/5900, RunningAvgSamplesPerSec=5.874788303180257, CurrSamplesPerSec=5.352282109839765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2915/5000 [9:49:29<6:37:10, 11.43s/it][2022-12-20 20:12:00,188] [INFO] [timer.py:197:stop] 0/5902, RunningAvgSamplesPerSec=5.874790439555649, CurrSamplesPerSec=5.338493035877214, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2916/5000 [9:49:40<6:32:47, 11.31s/it][2022-12-20 20:12:11,252] [INFO] [timer.py:197:stop] 0/5904, RunningAvgSamplesPerSec=5.874781919178066, CurrSamplesPerSec=5.305462279726399, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2917/5000 [9:49:51<6:29:22, 11.22s/it][2022-12-20 20:12:22,217] [INFO] [timer.py:197:stop] 0/5906, RunningAvgSamplesPerSec=5.874791483180663, CurrSamplesPerSec=5.335204250713688, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2918/5000 [9:50:02<6:27:22, 11.16s/it][2022-12-20 20:12:33,278] [INFO] [timer.py:197:stop] 0/5908, RunningAvgSamplesPerSec=5.874783358807379, CurrSamplesPerSec=5.3183357582982715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2919/5000 [9:50:13<6:25:49, 11.12s/it][2022-12-20 20:12:44,306] [INFO] [timer.py:197:stop] 0/5910, RunningAvgSamplesPerSec=5.874781700411822, CurrSamplesPerSec=5.323015503237727, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2920/5000 [9:50:24<6:24:42, 11.10s/it][2022-12-20 20:12:55,381] [INFO] [timer.py:197:stop] 0/5912, RunningAvgSamplesPerSec=5.8747712202171, CurrSamplesPerSec=5.287040626407139, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2921/5000 [9:50:35<6:24:02, 11.08s/it][2022-12-20 20:13:06,403] [INFO] [timer.py:197:stop] 0/5914, RunningAvgSamplesPerSec=5.874770505718344, CurrSamplesPerSec=5.312512933191664, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2922/5000 [9:50:46<6:23:09, 11.06s/it][2022-12-20 20:13:17,433] [INFO] [timer.py:197:stop] 0/5916, RunningAvgSamplesPerSec=5.87476816641745, CurrSamplesPerSec=5.300320777115527, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2923/5000 [9:50:57<6:22:37, 11.05s/it][2022-12-20 20:13:28,467] [INFO] [timer.py:197:stop] 0/5918, RunningAvgSamplesPerSec=5.874764854642008, CurrSamplesPerSec=5.295299953125821, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2924/5000 [9:51:08<6:22:19, 11.05s/it][2022-12-20 20:13:39,452] [INFO] [logging.py:68:log_dist] [Rank 0] step=2960, skipped=3, lr=[4.542222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:13:39,454] [INFO] [timer.py:197:stop] 0/5920, RunningAvgSamplesPerSec=5.874772209248993, CurrSamplesPerSec=5.345957366961097, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 58%|█████▊    | 2925/5000 [9:51:20<6:21:42, 11.04s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.542222222222223e-06, 'epoch': 71.34}

 58%|█████▊    | 2925/5000 [9:51:20<6:21:42, 11.04s/it][2022-12-20 20:13:50,522] [INFO] [timer.py:197:stop] 0/5922, RunningAvgSamplesPerSec=5.874763098552551, CurrSamplesPerSec=5.292787471110896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2926/5000 [9:51:31<6:21:53, 11.05s/it][2022-12-20 20:14:01,559] [INFO] [timer.py:197:stop] 0/5924, RunningAvgSamplesPerSec=5.874759759457521, CurrSamplesPerSec=5.324651462893609, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2927/5000 [9:51:42<6:21:39, 11.05s/it][2022-12-20 20:14:12,623] [INFO] [timer.py:197:stop] 0/5926, RunningAvgSamplesPerSec=5.87475105028978, CurrSamplesPerSec=5.305039101709108, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2928/5000 [9:51:53<6:21:13, 11.04s/it][2022-12-20 20:14:23,632] [INFO] [timer.py:197:stop] 0/5928, RunningAvgSamplesPerSec=5.874752551978114, CurrSamplesPerSec=5.316676508349269, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2929/5000 [9:52:04<6:20:34, 11.03s/it][2022-12-20 20:14:34,622] [INFO] [timer.py:197:stop] 0/5930, RunningAvgSamplesPerSec=5.874757505869775, CurrSamplesPerSec=5.321209239301193, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2930/5000 [9:52:15<6:20:12, 11.02s/it][2022-12-20 20:14:45,614] [INFO] [timer.py:197:stop] 0/5932, RunningAvgSamplesPerSec=5.874762211949463, CurrSamplesPerSec=5.335565653274986, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2931/5000 [9:52:26<6:20:03, 11.02s/it][2022-12-20 20:14:56,627] [INFO] [timer.py:197:stop] 0/5934, RunningAvgSamplesPerSec=5.874763046653146, CurrSamplesPerSec=5.3460261449082, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2932/5000 [9:52:37<6:19:38, 11.01s/it][2022-12-20 20:15:07,619] [INFO] [timer.py:197:stop] 0/5936, RunningAvgSamplesPerSec=5.87476734348689, CurrSamplesPerSec=5.35145453218098, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2933/5000 [9:52:48<6:18:46, 11.00s/it][2022-12-20 20:15:18,613] [INFO] [timer.py:197:stop] 0/5938, RunningAvgSamplesPerSec=5.874771674772431, CurrSamplesPerSec=5.3119469299735655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2934/5000 [9:52:59<6:18:37, 11.00s/it][2022-12-20 20:15:29,603] [INFO] [logging.py:68:log_dist] [Rank 0] step=2970, skipped=3, lr=[4.520000000000001e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:15:29,605] [INFO] [timer.py:197:stop] 0/5940, RunningAvgSamplesPerSec=5.874776328146612, CurrSamplesPerSec=5.316928194346642, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2935/5000 [9:53:10<6:18:52, 11.01s/it][2022-12-20 20:15:40,599] [INFO] [timer.py:197:stop] 0/5942, RunningAvgSamplesPerSec=5.874780488742046, CurrSamplesPerSec=5.3559589383896276, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2936/5000 [9:53:21<6:18:41, 11.01s/it][2022-12-20 20:15:51,620] [INFO] [timer.py:197:stop] 0/5944, RunningAvgSamplesPerSec=5.874779672772005, CurrSamplesPerSec=5.344396179126073, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▊    | 2937/5000 [9:53:32<6:18:51, 11.02s/it][2022-12-20 20:16:02,701] [INFO] [timer.py:197:stop] 0/5946, RunningAvgSamplesPerSec=5.874768220669111, CurrSamplesPerSec=5.311476473781591, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2938/5000 [9:53:43<6:18:58, 11.03s/it][2022-12-20 20:16:13,754] [INFO] [timer.py:197:stop] 0/5948, RunningAvgSamplesPerSec=5.874761910084056, CurrSamplesPerSec=5.305818825001748, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2939/5000 [9:53:54<6:19:10, 11.04s/it][2022-12-20 20:16:24,798] [INFO] [timer.py:197:stop] 0/5950, RunningAvgSamplesPerSec=5.874757065495042, CurrSamplesPerSec=5.3242602784617015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2940/5000 [9:54:05<6:18:57, 11.04s/it][2022-12-20 20:16:35,863] [INFO] [timer.py:197:stop] 0/5952, RunningAvgSamplesPerSec=5.874748328657431, CurrSamplesPerSec=5.297274525460245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2941/5000 [9:54:16<6:18:34, 11.03s/it][2022-12-20 20:16:46,843] [INFO] [timer.py:197:stop] 0/5954, RunningAvgSamplesPerSec=5.874755061416279, CurrSamplesPerSec=5.3316952122298495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2942/5000 [9:54:27<6:18:20, 11.03s/it][2022-12-20 20:16:57,861] [INFO] [timer.py:197:stop] 0/5956, RunningAvgSamplesPerSec=5.874754928720397, CurrSamplesPerSec=5.339744211499144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2943/5000 [9:54:38<6:18:08, 11.03s/it][2022-12-20 20:17:08,882] [INFO] [timer.py:197:stop] 0/5958, RunningAvgSamplesPerSec=5.874754514321736, CurrSamplesPerSec=5.347652843880279, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2944/5000 [9:54:49<6:17:51, 11.03s/it][2022-12-20 20:17:19,912] [INFO] [logging.py:68:log_dist] [Rank 0] step=2980, skipped=3, lr=[4.497777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:17:19,914] [INFO] [timer.py:197:stop] 0/5960, RunningAvgSamplesPerSec=5.874751959958992, CurrSamplesPerSec=5.33694617808222, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2945/5000 [9:55:00<6:17:19, 11.02s/it][2022-12-20 20:17:30,910] [INFO] [timer.py:197:stop] 0/5962, RunningAvgSamplesPerSec=5.874755915365653, CurrSamplesPerSec=5.334873008038894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2946/5000 [9:55:11<6:17:18, 11.02s/it][2022-12-20 20:17:41,956] [INFO] [timer.py:197:stop] 0/5964, RunningAvgSamplesPerSec=5.87475062471221, CurrSamplesPerSec=5.321819209562847, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2947/5000 [9:55:22<6:16:27, 11.00s/it][2022-12-20 20:17:52,913] [INFO] [timer.py:197:stop] 0/5966, RunningAvgSamplesPerSec=5.874761532254081, CurrSamplesPerSec=5.322460557032565, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2948/5000 [9:55:33<6:16:35, 11.01s/it][2022-12-20 20:18:03,980] [INFO] [timer.py:197:stop] 0/5968, RunningAvgSamplesPerSec=5.8747528738851384, CurrSamplesPerSec=5.292777870119689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2949/5000 [9:55:44<6:16:06, 11.00s/it][2022-12-20 20:18:14,893] [INFO] [timer.py:197:stop] 0/5970, RunningAvgSamplesPerSec=5.874771870257053, CurrSamplesPerSec=5.355048818866976, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2950/5000 [9:55:55<6:15:56, 11.00s/it]
                                                       
{'loss': 0.0001, 'learning_rate': 4.486666666666667e-06, 'epoch': 71.94}

 59%|█████▉    | 2950/5000 [9:55:55<6:15:56, 11.00s/it][2022-12-20 20:18:25,896] [INFO] [timer.py:197:stop] 0/5972, RunningAvgSamplesPerSec=5.874774743899718, CurrSamplesPerSec=5.356706882558939, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2951/5000 [9:56:06<6:15:34, 11.00s/it][2022-12-20 20:18:36,871] [INFO] [timer.py:197:stop] 0/5974, RunningAvgSamplesPerSec=5.874782800833587, CurrSamplesPerSec=5.366402211017528, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2952/5000 [9:56:17<6:15:18, 11.00s/it][2022-12-20 20:18:46,982] [INFO] [timer.py:197:stop] 0/5976, RunningAvgSamplesPerSec=5.874944533486637, CurrSamplesPerSec=6.279710194562957, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 20:18:57,978] [INFO] [timer.py:197:stop] 0/5978, RunningAvgSamplesPerSec=5.874947750348053, CurrSamplesPerSec=5.319260002395333, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2953/5000 [9:56:33<7:07:50, 12.54s/it][2022-12-20 20:19:08,966] [INFO] [logging.py:68:log_dist] [Rank 0] step=2990, skipped=3, lr=[4.475555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:19:08,968] [INFO] [timer.py:197:stop] 0/5980, RunningAvgSamplesPerSec=5.874952473030796, CurrSamplesPerSec=5.349581584054581, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2954/5000 [9:56:44<6:51:46, 12.08s/it][2022-12-20 20:19:19,929] [INFO] [timer.py:197:stop] 0/5982, RunningAvgSamplesPerSec=5.874962476942582, CurrSamplesPerSec=5.3563727507902, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2955/5000 [9:56:55<6:40:10, 11.74s/it][2022-12-20 20:19:30,851] [INFO] [timer.py:197:stop] 0/5984, RunningAvgSamplesPerSec=5.874979645858327, CurrSamplesPerSec=5.370755807623169, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2956/5000 [9:57:06<6:31:36, 11.50s/it][2022-12-20 20:19:41,853] [INFO] [timer.py:197:stop] 0/5986, RunningAvgSamplesPerSec=5.874982423680192, CurrSamplesPerSec=5.3164063146762155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2957/5000 [9:57:17<6:26:22, 11.35s/it][2022-12-20 20:19:52,834] [INFO] [timer.py:197:stop] 0/5988, RunningAvgSamplesPerSec=5.874988703275025, CurrSamplesPerSec=5.3557657338357245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2958/5000 [9:57:28<6:22:27, 11.24s/it][2022-12-20 20:20:03,840] [INFO] [timer.py:197:stop] 0/5990, RunningAvgSamplesPerSec=5.874990853164031, CurrSamplesPerSec=5.320454723836691, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2959/5000 [9:57:39<6:19:53, 11.17s/it][2022-12-20 20:20:14,878] [INFO] [timer.py:197:stop] 0/5992, RunningAvgSamplesPerSec=5.874986959762335, CurrSamplesPerSec=5.30235565344699, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2960/5000 [9:57:50<6:18:23, 11.13s/it][2022-12-20 20:20:25,875] [INFO] [timer.py:197:stop] 0/5994, RunningAvgSamplesPerSec=5.8749905872983765, CurrSamplesPerSec=5.338349711655825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2961/5000 [9:58:01<6:16:51, 11.09s/it][2022-12-20 20:20:36,919] [INFO] [timer.py:197:stop] 0/5996, RunningAvgSamplesPerSec=5.874985603502789, CurrSamplesPerSec=5.325078832100223, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2962/5000 [9:58:12<6:16:12, 11.08s/it][2022-12-20 20:20:47,905] [INFO] [timer.py:197:stop] 0/5998, RunningAvgSamplesPerSec=5.874991275074586, CurrSamplesPerSec=5.368897246159506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2963/5000 [9:58:23<6:15:06, 11.05s/it][2022-12-20 20:20:58,907] [INFO] [logging.py:68:log_dist] [Rank 0] step=3000, skipped=3, lr=[4.453333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:20:58,909] [INFO] [timer.py:197:stop] 0/6000, RunningAvgSamplesPerSec=5.874993508244673, CurrSamplesPerSec=5.311423085002095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2964/5000 [9:58:34<6:14:27, 11.04s/it][2022-12-20 20:21:09,903] [INFO] [timer.py:197:stop] 0/6002, RunningAvgSamplesPerSec=5.874997682394482, CurrSamplesPerSec=5.32820115105702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2965/5000 [9:58:45<6:13:51, 11.02s/it][2022-12-20 20:21:20,945] [INFO] [timer.py:197:stop] 0/6004, RunningAvgSamplesPerSec=5.874993009041949, CurrSamplesPerSec=5.3093995170034, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2966/5000 [9:58:56<6:13:52, 11.03s/it][2022-12-20 20:21:31,974] [INFO] [timer.py:197:stop] 0/6006, RunningAvgSamplesPerSec=5.874990890584703, CurrSamplesPerSec=5.318863075555646, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2967/5000 [9:59:07<6:13:41, 11.03s/it][2022-12-20 20:21:43,014] [INFO] [timer.py:197:stop] 0/6008, RunningAvgSamplesPerSec=5.874986751495474, CurrSamplesPerSec=5.300674328907552, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2968/5000 [9:59:18<6:13:37, 11.03s/it][2022-12-20 20:21:54,113] [INFO] [timer.py:197:stop] 0/6010, RunningAvgSamplesPerSec=5.874972077451004, CurrSamplesPerSec=5.234501525994954, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2969/5000 [9:59:29<6:14:07, 11.05s/it][2022-12-20 20:22:05,142] [INFO] [timer.py:197:stop] 0/6012, RunningAvgSamplesPerSec=5.874969949476336, CurrSamplesPerSec=5.294270825115543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2970/5000 [9:59:40<6:13:41, 11.05s/it][2022-12-20 20:22:16,185] [INFO] [timer.py:197:stop] 0/6014, RunningAvgSamplesPerSec=5.8749651629196205, CurrSamplesPerSec=5.315109430459475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2971/5000 [9:59:51<6:13:29, 11.04s/it][2022-12-20 20:22:27,221] [INFO] [timer.py:197:stop] 0/6016, RunningAvgSamplesPerSec=5.874961833134361, CurrSamplesPerSec=5.3111216897424605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2972/5000 [10:00:02<6:13:12, 11.04s/it][2022-12-20 20:22:38,162] [INFO] [timer.py:197:stop] 0/6018, RunningAvgSamplesPerSec=5.874975347528325, CurrSamplesPerSec=5.4081553397432325, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2973/5000 [10:00:13<6:12:00, 11.01s/it][2022-12-20 20:22:49,132] [INFO] [logging.py:68:log_dist] [Rank 0] step=3010, skipped=3, lr=[4.431111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:22:49,134] [INFO] [timer.py:197:stop] 0/6020, RunningAvgSamplesPerSec=5.874983345465486, CurrSamplesPerSec=5.36519255818966, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 59%|█████▉    | 2974/5000 [10:00:24<6:11:25, 11.00s/it][2022-12-20 20:23:00,138] [INFO] [timer.py:197:stop] 0/6022, RunningAvgSamplesPerSec=5.874985678743617, CurrSamplesPerSec=5.358299444503174, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2975/5000 [10:00:35<6:11:17, 11.00s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 4.4288888888888895e-06, 'epoch': 72.55}

 60%|█████▉    | 2975/5000 [10:00:35<6:11:17, 11.00s/it][2022-12-20 20:23:11,080] [INFO] [timer.py:197:stop] 0/6024, RunningAvgSamplesPerSec=5.874999499314156, CurrSamplesPerSec=5.329793102681461, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2976/5000 [10:00:46<6:10:30, 10.98s/it][2022-12-20 20:23:22,148] [INFO] [timer.py:197:stop] 0/6026, RunningAvgSamplesPerSec=5.874990329985133, CurrSamplesPerSec=5.304369452464169, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2977/5000 [10:00:57<6:11:10, 11.01s/it][2022-12-20 20:23:33,138] [INFO] [timer.py:197:stop] 0/6028, RunningAvgSamplesPerSec=5.874995148556201, CurrSamplesPerSec=5.336265691846711, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2978/5000 [10:01:08<6:10:48, 11.00s/it][2022-12-20 20:23:44,175] [INFO] [timer.py:197:stop] 0/6030, RunningAvgSamplesPerSec=5.874991843398066, CurrSamplesPerSec=5.315187730840546, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2979/5000 [10:01:19<6:10:57, 11.01s/it][2022-12-20 20:23:55,219] [INFO] [timer.py:197:stop] 0/6032, RunningAvgSamplesPerSec=5.874987017091406, CurrSamplesPerSec=5.3069004989923165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2980/5000 [10:01:30<6:11:05, 11.02s/it][2022-12-20 20:24:06,258] [INFO] [timer.py:197:stop] 0/6034, RunningAvgSamplesPerSec=5.874983300693197, CurrSamplesPerSec=5.3164844426820235, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2981/5000 [10:01:41<6:11:04, 11.03s/it][2022-12-20 20:24:17,223] [INFO] [timer.py:197:stop] 0/6036, RunningAvgSamplesPerSec=5.874992357534083, CurrSamplesPerSec=5.356019424538606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2982/5000 [10:01:52<6:10:15, 11.01s/it][2022-12-20 20:24:28,250] [INFO] [timer.py:197:stop] 0/6038, RunningAvgSamplesPerSec=5.874990588761148, CurrSamplesPerSec=5.320033790040056, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2983/5000 [10:02:03<6:10:15, 11.01s/it][2022-12-20 20:24:39,196] [INFO] [logging.py:68:log_dist] [Rank 0] step=3020, skipped=3, lr=[4.408888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:24:39,198] [INFO] [timer.py:197:stop] 0/6040, RunningAvgSamplesPerSec=5.875002771617135, CurrSamplesPerSec=5.355393042308162, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2984/5000 [10:02:14<6:09:24, 10.99s/it][2022-12-20 20:24:50,168] [INFO] [timer.py:197:stop] 0/6042, RunningAvgSamplesPerSec=5.8750110276213015, CurrSamplesPerSec=5.369174520047398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2985/5000 [10:02:25<6:08:58, 10.99s/it][2022-12-20 20:25:01,208] [INFO] [timer.py:197:stop] 0/6044, RunningAvgSamplesPerSec=5.875006816225193, CurrSamplesPerSec=5.305942997900634, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2986/5000 [10:02:36<6:09:19, 11.00s/it][2022-12-20 20:25:12,203] [INFO] [timer.py:197:stop] 0/6046, RunningAvgSamplesPerSec=5.875010622193367, CurrSamplesPerSec=5.310001740754221, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2987/5000 [10:02:47<6:09:04, 11.00s/it][2022-12-20 20:25:23,231] [INFO] [timer.py:197:stop] 0/6048, RunningAvgSamplesPerSec=5.875008800570498, CurrSamplesPerSec=5.3236824783347405, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2988/5000 [10:02:58<6:09:09, 11.01s/it][2022-12-20 20:25:34,286] [INFO] [timer.py:197:stop] 0/6050, RunningAvgSamplesPerSec=5.875002121418948, CurrSamplesPerSec=5.308097651508671, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2989/5000 [10:03:09<6:09:26, 11.02s/it][2022-12-20 20:25:45,274] [INFO] [timer.py:197:stop] 0/6052, RunningAvgSamplesPerSec=5.875007438801493, CurrSamplesPerSec=5.3362031051645795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2990/5000 [10:03:20<6:08:54, 11.01s/it][2022-12-20 20:25:56,339] [INFO] [timer.py:197:stop] 0/6054, RunningAvgSamplesPerSec=5.874999018904711, CurrSamplesPerSec=5.308771392175493, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2991/5000 [10:03:31<6:09:15, 11.03s/it][2022-12-20 20:26:07,263] [INFO] [timer.py:197:stop] 0/6056, RunningAvgSamplesPerSec=5.87501571571107, CurrSamplesPerSec=5.356558089367755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2992/5000 [10:03:42<6:08:01, 11.00s/it][2022-12-20 20:26:18,268] [INFO] [timer.py:197:stop] 0/6058, RunningAvgSamplesPerSec=5.875017984622778, CurrSamplesPerSec=5.313190739848801, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2993/5000 [10:03:53<6:07:55, 11.00s/it][2022-12-20 20:26:28,398] [INFO] [logging.py:68:log_dist] [Rank 0] step=3030, skipped=3, lr=[4.3866666666666665e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:26:28,400] [INFO] [timer.py:197:stop] 0/6060, RunningAvgSamplesPerSec=5.875174020225311, CurrSamplesPerSec=5.311957021099694, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2994/5000 [10:04:08<6:48:39, 12.22s/it][2022-12-20 20:26:39,442] [INFO] [timer.py:197:stop] 0/6062, RunningAvgSamplesPerSec=5.875169564795225, CurrSamplesPerSec=5.306426530525911, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2995/5000 [10:04:19<6:36:26, 11.86s/it][2022-12-20 20:26:50,464] [INFO] [timer.py:197:stop] 0/6064, RunningAvgSamplesPerSec=5.8751683152183585, CurrSamplesPerSec=5.307507402935565, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2996/5000 [10:04:31<6:28:03, 11.62s/it][2022-12-20 20:27:01,500] [INFO] [timer.py:197:stop] 0/6066, RunningAvgSamplesPerSec=5.875164760788702, CurrSamplesPerSec=5.317560568404629, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2997/5000 [10:04:41<6:21:25, 11.43s/it][2022-12-20 20:27:12,479] [INFO] [timer.py:197:stop] 0/6068, RunningAvgSamplesPerSec=5.875171212836504, CurrSamplesPerSec=5.313532127144026, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2998/5000 [10:04:53<6:17:19, 11.31s/it][2022-12-20 20:27:23,512] [INFO] [timer.py:197:stop] 0/6070, RunningAvgSamplesPerSec=5.875168091074225, CurrSamplesPerSec=5.315863693751322, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|█████▉    | 2999/5000 [10:05:04<6:14:04, 11.22s/it][2022-12-20 20:27:34,534] [INFO] [timer.py:197:stop] 0/6072, RunningAvgSamplesPerSec=5.875167211684567, CurrSamplesPerSec=5.298936120484576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3000/5000 [10:05:15<6:12:15, 11.17s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 4.3733333333333335e-06, 'epoch': 73.17}

 60%|██████    | 3000/5000 [10:05:15<6:12:15, 11.17s/it][INFO|trainer.py:2955] 2022-12-20 20:27:39,491 >> ***** Running Evaluation *****
[INFO|trainer.py:2957] 2022-12-20 20:27:39,492 >>   Num examples = 1325
[INFO|trainer.py:2960] 2022-12-20 20:27:39,492 >>   Batch size = 8


  0%|          | 0/166 [00:00<?, ?it/s]

  1%|          | 2/166 [00:05<07:53,  2.89s/it]

  2%|▏         | 3/166 [00:11<11:09,  4.11s/it]

  2%|▏         | 4/166 [00:17<12:56,  4.79s/it]

  3%|▎         | 5/166 [00:24<15:07,  5.64s/it]

  4%|▎         | 6/166 [00:32<16:32,  6.20s/it]

  4%|▍         | 7/166 [00:38<16:32,  6.24s/it]

  5%|▍         | 8/166 [00:44<16:08,  6.13s/it]

  5%|▌         | 9/166 [00:50<15:52,  6.07s/it]

  6%|▌         | 10/166 [00:56<15:32,  5.98s/it]

  7%|▋         | 11/166 [01:01<15:09,  5.87s/it]

  7%|▋         | 12/166 [01:07<15:09,  5.91s/it]

  8%|▊         | 13/166 [01:14<15:30,  6.08s/it]

  8%|▊         | 14/166 [01:21<16:04,  6.34s/it]

  9%|▉         | 15/166 [01:26<15:27,  6.14s/it]

 10%|▉         | 16/166 [01:34<16:15,  6.50s/it]

 10%|█         | 17/166 [01:40<16:18,  6.57s/it]

 11%|█         | 18/166 [01:47<16:00,  6.49s/it]

 11%|█▏        | 19/166 [01:52<15:27,  6.31s/it]

 12%|█▏        | 20/166 [02:03<18:32,  7.62s/it]

 13%|█▎        | 21/166 [02:10<18:07,  7.50s/it]

 13%|█▎        | 22/166 [02:18<17:52,  7.45s/it]

 14%|█▍        | 23/166 [02:25<17:50,  7.49s/it]

 14%|█▍        | 24/166 [02:32<17:00,  7.19s/it]

 15%|█▌        | 25/166 [02:39<16:54,  7.20s/it]

 16%|█▌        | 26/166 [02:46<16:41,  7.15s/it]

 16%|█▋        | 27/166 [02:53<16:44,  7.22s/it]

 17%|█▋        | 28/166 [02:59<15:47,  6.87s/it]

 17%|█▋        | 29/166 [03:08<16:36,  7.27s/it]

 18%|█▊        | 30/166 [03:15<16:12,  7.15s/it]

 19%|█▊        | 31/166 [03:22<16:10,  7.19s/it]

 19%|█▉        | 32/166 [03:30<16:23,  7.34s/it]

 20%|█▉        | 33/166 [03:37<16:18,  7.36s/it]

 20%|██        | 34/166 [03:43<15:34,  7.08s/it]

 21%|██        | 35/166 [03:51<16:01,  7.34s/it]

 22%|██▏       | 36/166 [03:59<15:58,  7.37s/it]

 22%|██▏       | 37/166 [04:05<15:16,  7.10s/it]

 23%|██▎       | 38/166 [04:14<16:04,  7.53s/it]

 23%|██▎       | 39/166 [04:21<15:31,  7.33s/it]

 24%|██▍       | 40/166 [04:28<15:37,  7.44s/it]

 25%|██▍       | 41/166 [04:35<15:02,  7.22s/it]

 25%|██▌       | 42/166 [04:42<14:35,  7.06s/it]

 26%|██▌       | 43/166 [04:48<14:05,  6.87s/it]

 27%|██▋       | 44/166 [04:57<14:58,  7.37s/it]

 27%|██▋       | 45/166 [05:05<15:17,  7.58s/it]

 28%|██▊       | 46/166 [05:15<16:37,  8.31s/it]

 28%|██▊       | 47/166 [05:23<16:25,  8.28s/it]

 29%|██▉       | 48/166 [05:30<15:35,  7.93s/it]

 30%|██▉       | 49/166 [05:36<14:15,  7.31s/it]

 30%|███       | 50/166 [05:44<14:50,  7.68s/it]

 31%|███       | 51/166 [05:52<14:34,  7.61s/it]

 31%|███▏      | 52/166 [06:00<14:28,  7.61s/it]

 32%|███▏      | 53/166 [06:07<14:04,  7.48s/it]

 33%|███▎      | 54/166 [06:15<14:13,  7.62s/it]

 33%|███▎      | 55/166 [06:21<13:09,  7.11s/it]

 34%|███▎      | 56/166 [06:27<12:31,  6.84s/it]

 34%|███▍      | 57/166 [06:35<13:07,  7.23s/it]

 35%|███▍      | 58/166 [06:40<12:06,  6.73s/it]

 36%|███▌      | 59/166 [06:47<12:07,  6.80s/it]

 36%|███▌      | 60/166 [06:55<12:10,  6.89s/it]

 37%|███▋      | 61/166 [07:04<13:13,  7.56s/it]

 37%|███▋      | 62/166 [07:11<12:59,  7.49s/it]

 38%|███▊      | 63/166 [07:18<12:41,  7.39s/it]

 39%|███▊      | 64/166 [07:27<13:28,  7.93s/it]

 39%|███▉      | 65/166 [07:36<13:37,  8.09s/it]

 40%|███▉      | 66/166 [07:43<12:52,  7.72s/it]

 40%|████      | 67/166 [07:50<12:21,  7.49s/it]

 41%|████      | 68/166 [07:55<11:04,  6.78s/it]

 42%|████▏     | 69/166 [08:02<11:00,  6.81s/it]

 42%|████▏     | 70/166 [08:10<11:28,  7.17s/it]

 43%|████▎     | 71/166 [08:18<11:41,  7.39s/it]

 43%|████▎     | 72/166 [08:24<11:08,  7.12s/it]

 44%|████▍     | 73/166 [08:30<10:42,  6.91s/it]

 45%|████▍     | 74/166 [08:37<10:19,  6.73s/it]

 45%|████▌     | 75/166 [08:43<09:59,  6.59s/it]

 46%|████▌     | 76/166 [08:50<10:13,  6.81s/it]

 46%|████▋     | 77/166 [08:56<09:38,  6.50s/it]

 47%|████▋     | 78/166 [09:03<09:48,  6.68s/it]

 48%|████▊     | 79/166 [09:11<10:14,  7.07s/it]

 48%|████▊     | 80/166 [09:19<10:39,  7.43s/it]

 49%|████▉     | 81/166 [09:27<10:42,  7.56s/it]

 49%|████▉     | 82/166 [09:34<10:19,  7.37s/it]

 50%|█████     | 83/166 [09:40<09:21,  6.77s/it]

 51%|█████     | 84/166 [09:46<09:06,  6.67s/it]

 51%|█████     | 85/166 [09:53<09:14,  6.85s/it]

 52%|█████▏    | 86/166 [10:02<09:42,  7.28s/it]

 52%|█████▏    | 87/166 [10:09<09:42,  7.37s/it]

 53%|█████▎    | 88/166 [10:16<09:19,  7.18s/it]

 54%|█████▎    | 89/166 [10:23<09:16,  7.23s/it]

 54%|█████▍    | 90/166 [10:30<08:53,  7.02s/it]

 55%|█████▍    | 91/166 [10:37<08:39,  6.93s/it]

 55%|█████▌    | 92/166 [10:43<08:11,  6.65s/it]

 56%|█████▌    | 93/166 [10:51<08:36,  7.08s/it]

 57%|█████▋    | 94/166 [11:01<09:36,  8.01s/it]

 57%|█████▋    | 95/166 [11:06<08:30,  7.18s/it]

 58%|█████▊    | 96/166 [11:13<08:08,  6.98s/it]

 58%|█████▊    | 97/166 [11:23<09:09,  7.96s/it]

 59%|█████▉    | 98/166 [11:33<09:40,  8.54s/it]

 60%|█████▉    | 99/166 [11:40<09:10,  8.22s/it]

 60%|██████    | 100/166 [11:46<08:22,  7.61s/it]

 61%|██████    | 101/166 [11:53<07:55,  7.31s/it]

 61%|██████▏   | 102/166 [12:02<08:17,  7.78s/it]

 62%|██████▏   | 103/166 [12:11<08:32,  8.13s/it]

 63%|██████▎   | 104/166 [12:18<08:08,  7.88s/it]

 63%|██████▎   | 105/166 [12:26<08:05,  7.96s/it]

 64%|██████▍   | 106/166 [12:36<08:32,  8.55s/it]

 64%|██████▍   | 107/166 [12:46<08:41,  8.84s/it]

 65%|██████▌   | 108/166 [12:51<07:34,  7.83s/it]

 66%|██████▌   | 109/166 [12:58<07:01,  7.39s/it]

 66%|██████▋   | 110/166 [13:05<06:55,  7.41s/it]

 67%|██████▋   | 111/166 [13:11<06:24,  6.99s/it]

 67%|██████▋   | 112/166 [13:18<06:23,  7.09s/it]

 68%|██████▊   | 113/166 [13:27<06:39,  7.53s/it]

 69%|██████▊   | 114/166 [13:34<06:30,  7.51s/it]

 69%|██████▉   | 115/166 [13:41<06:05,  7.17s/it]

 70%|██████▉   | 116/166 [13:47<05:44,  6.90s/it]

 70%|███████   | 117/166 [13:54<05:47,  7.09s/it]

 71%|███████   | 118/166 [14:02<05:50,  7.29s/it]

 72%|███████▏  | 119/166 [14:10<05:48,  7.42s/it]

 72%|███████▏  | 120/166 [14:18<05:52,  7.66s/it]

 73%|███████▎  | 121/166 [14:27<05:54,  7.87s/it]

 73%|███████▎  | 122/166 [14:34<05:43,  7.80s/it]

 74%|███████▍  | 123/166 [14:42<05:29,  7.67s/it]

 75%|███████▍  | 124/166 [14:48<05:10,  7.40s/it]

 75%|███████▌  | 125/166 [14:56<05:04,  7.42s/it]

 76%|███████▌  | 126/166 [15:04<05:09,  7.74s/it]

 77%|███████▋  | 127/166 [15:09<04:29,  6.92s/it]

 77%|███████▋  | 128/166 [15:19<04:59,  7.88s/it]

 78%|███████▊  | 129/166 [15:27<04:51,  7.88s/it]

 78%|███████▊  | 130/166 [15:34<04:32,  7.58s/it]

 79%|███████▉  | 131/166 [15:41<04:19,  7.43s/it]

 80%|███████▉  | 132/166 [15:49<04:19,  7.63s/it]

 80%|████████  | 133/166 [15:57<04:13,  7.68s/it]

 81%|████████  | 134/166 [16:04<03:54,  7.34s/it]

 81%|████████▏ | 135/166 [16:10<03:36,  6.99s/it]

 82%|████████▏ | 136/166 [16:20<04:01,  8.03s/it]

 83%|████████▎ | 137/166 [16:29<04:01,  8.33s/it]

 83%|████████▎ | 138/166 [16:39<04:08,  8.86s/it]

 84%|████████▎ | 139/166 [16:48<03:54,  8.69s/it]

 84%|████████▍ | 140/166 [16:54<03:27,  7.98s/it]

 85%|████████▍ | 141/166 [17:00<03:04,  7.37s/it]

 86%|████████▌ | 142/166 [17:07<02:55,  7.31s/it]

 86%|████████▌ | 143/166 [17:13<02:35,  6.77s/it]

 87%|████████▋ | 144/166 [17:21<02:42,  7.38s/it]

 87%|████████▋ | 145/166 [17:31<02:49,  8.09s/it]

 88%|████████▊ | 146/166 [17:41<02:49,  8.47s/it]

 89%|████████▊ | 147/166 [17:49<02:40,  8.47s/it]

 89%|████████▉ | 148/166 [17:57<02:32,  8.45s/it]

 90%|████████▉ | 149/166 [18:07<02:27,  8.68s/it]

 90%|█████████ | 150/166 [18:16<02:23,  8.98s/it]

 91%|█████████ | 151/166 [18:24<02:10,  8.69s/it]

 92%|█████████▏| 152/166 [18:30<01:49,  7.81s/it]

 92%|█████████▏| 153/166 [18:37<01:36,  7.41s/it]

 93%|█████████▎| 154/166 [18:44<01:28,  7.39s/it]

 93%|█████████▎| 155/166 [18:52<01:22,  7.51s/it]

 94%|█████████▍| 156/166 [18:58<01:10,  7.02s/it]

 95%|█████████▍| 157/166 [19:04<01:02,  6.92s/it]

 95%|█████████▌| 158/166 [19:10<00:52,  6.53s/it]

 96%|█████████▌| 159/166 [19:15<00:41,  6.00s/it]

 96%|█████████▋| 160/166 [19:20<00:34,  5.71s/it]

 97%|█████████▋| 161/166 [19:26<00:29,  5.83s/it]

 98%|█████████▊| 162/166 [19:33<00:24,  6.16s/it]

 98%|█████████▊| 163/166 [19:40<00:19,  6.46s/it]

 99%|█████████▉| 164/166 [19:46<00:12,  6.37s/it]

 99%|█████████▉| 165/166 [19:52<00:06,  6.24s/it]

100%|██████████| 166/166 [19:56<00:00,  5.62s/it]
                                                        

                                                 
{'eval_loss': 0.275634765625, 'eval_wer': 8.451041417285133, 'eval_runtime': 1273.9374, 'eval_samples_per_second': 1.04, 'eval_steps_per_second': 0.13, 'epoch': 73.17}

 60%|██████    | 3000/5000 [10:26:29<6:12:15, 11.17s/it]

100%|██████████| 166/166 [21:08<00:00,  5.62s/it]

                                                 [INFO|trainer.py:2700] 2022-12-20 20:48:53,431 >> Saving model checkpoint to ./checkpoint-3000
[INFO|configuration_utils.py:447] 2022-12-20 20:48:53,432 >> Configuration saved in ./checkpoint-3000/config.json
[INFO|modeling_utils.py:1680] 2022-12-20 20:48:56,985 >> Model weights saved in ./checkpoint-3000/pytorch_model.bin
[INFO|feature_extraction_utils.py:368] 2022-12-20 20:48:57,000 >> Feature extractor saved in ./checkpoint-3000/preprocessor_config.json
[2022-12-20 20:48:57,001] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step3036 is begin to save!
/home/milan/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
[2022-12-20 20:48:57,011] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-3000/global_step3036/mp_rank_00_model_states.pt
[2022-12-20 20:48:57,011] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-3000/global_step3036/mp_rank_00_model_states.pt...
[2022-12-20 20:49:00,512] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-3000/global_step3036/mp_rank_00_model_states.pt.
[2022-12-20 20:49:00,513] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-3000/global_step3036/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2022-12-20 20:49:16,059] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-3000/global_step3036/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2022-12-20 20:49:16,059] [INFO] [engine.py:3269:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-3000/global_step3036/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2022-12-20 20:49:16,059] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3036 is ready now!
[INFO|feature_extraction_utils.py:368] 2022-12-20 20:49:20,274 >> Feature extractor saved in ./preprocessor_config.json
[2022-12-20 20:51:21,162] [INFO] [timer.py:197:stop] 0/6074, RunningAvgSamplesPerSec=5.875122686635279, CurrSamplesPerSec=5.118360486106803, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3001/5000 [10:29:01<241:59:39, 435.81s/it][2022-12-20 20:51:32,145] [INFO] [timer.py:197:stop] 0/6076, RunningAvgSamplesPerSec=5.875130666194641, CurrSamplesPerSec=5.367977786706339, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3002/5000 [10:29:12<171:08:17, 308.36s/it][2022-12-20 20:51:43,113] [INFO] [timer.py:197:stop] 0/6078, RunningAvgSamplesPerSec=5.875139350458805, CurrSamplesPerSec=5.370491693602941, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3003/5000 [10:29:23<121:33:41, 219.14s/it][2022-12-20 20:51:54,116] [INFO] [logging.py:68:log_dist] [Rank 0] step=3040, skipped=3, lr=[4.364444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:51:54,117] [INFO] [timer.py:197:stop] 0/6080, RunningAvgSamplesPerSec=5.875141544801326, CurrSamplesPerSec=5.335915225334919, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3004/5000 [10:29:34<86:52:34, 156.69s/it] [2022-12-20 20:52:05,113] [INFO] [timer.py:197:stop] 0/6082, RunningAvgSamplesPerSec=5.87514513650894, CurrSamplesPerSec=5.318208686681358, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3005/5000 [10:29:45<62:36:52, 112.99s/it][2022-12-20 20:52:16,146] [INFO] [timer.py:197:stop] 0/6084, RunningAvgSamplesPerSec=5.875142082756348, CurrSamplesPerSec=5.305457246489068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3006/5000 [10:29:56<45:38:29, 82.40s/it] [2022-12-20 20:52:27,173] [INFO] [timer.py:197:stop] 0/6086, RunningAvgSamplesPerSec=5.875140137022045, CurrSamplesPerSec=5.3097983935669255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3007/5000 [10:30:07<33:45:29, 60.98s/it][2022-12-20 20:52:38,135] [INFO] [timer.py:197:stop] 0/6088, RunningAvgSamplesPerSec=5.87514987678892, CurrSamplesPerSec=5.334614743635069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3008/5000 [10:30:18<25:26:32, 45.98s/it][2022-12-20 20:52:49,105] [INFO] [timer.py:197:stop] 0/6090, RunningAvgSamplesPerSec=5.875158318778088, CurrSamplesPerSec=5.34843043902184, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3009/5000 [10:30:29<19:37:27, 35.48s/it][2022-12-20 20:53:00,099] [INFO] [timer.py:197:stop] 0/6092, RunningAvgSamplesPerSec=5.875162612941647, CurrSamplesPerSec=5.345334613176732, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3010/5000 [10:30:40<15:33:13, 28.14s/it][2022-12-20 20:53:11,101] [INFO] [timer.py:197:stop] 0/6094, RunningAvgSamplesPerSec=5.875165199593417, CurrSamplesPerSec=5.3400121085292485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3011/5000 [10:30:51<12:41:56, 22.98s/it][2022-12-20 20:53:22,063] [INFO] [timer.py:197:stop] 0/6096, RunningAvgSamplesPerSec=5.875174976796689, CurrSamplesPerSec=5.340345689587291, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3012/5000 [10:31:02<10:42:00, 19.38s/it][2022-12-20 20:53:32,978] [INFO] [timer.py:197:stop] 0/6098, RunningAvgSamplesPerSec=5.875192878929816, CurrSamplesPerSec=5.378011358686893, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3013/5000 [10:31:13<9:18:14, 16.86s/it] [2022-12-20 20:53:44,006] [INFO] [logging.py:68:log_dist] [Rank 0] step=3050, skipped=3, lr=[4.3422222222222225e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:53:44,008] [INFO] [timer.py:197:stop] 0/6100, RunningAvgSamplesPerSec=5.875190564344091, CurrSamplesPerSec=5.33053439134569, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3014/5000 [10:31:24<8:20:05, 15.11s/it][2022-12-20 20:53:55,007] [INFO] [timer.py:197:stop] 0/6102, RunningAvgSamplesPerSec=5.875193671286669, CurrSamplesPerSec=5.358543320296199, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3015/5000 [10:31:35<7:39:05, 13.88s/it][2022-12-20 20:54:06,004] [INFO] [timer.py:197:stop] 0/6104, RunningAvgSamplesPerSec=5.875197024987366, CurrSamplesPerSec=5.363621403281703, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3016/5000 [10:31:46<7:09:40, 12.99s/it][2022-12-20 20:54:16,953] [INFO] [timer.py:197:stop] 0/6106, RunningAvgSamplesPerSec=5.875208802025289, CurrSamplesPerSec=5.34943446568799, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3017/5000 [10:31:57<6:49:33, 12.39s/it][2022-12-20 20:54:27,950] [INFO] [timer.py:197:stop] 0/6108, RunningAvgSamplesPerSec=5.875212103648999, CurrSamplesPerSec=5.341567546980284, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3018/5000 [10:32:08<6:35:41, 11.98s/it][2022-12-20 20:54:39,012] [INFO] [timer.py:197:stop] 0/6110, RunningAvgSamplesPerSec=5.875203971792002, CurrSamplesPerSec=5.29928927834003, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3019/5000 [10:32:19<6:25:54, 11.69s/it][2022-12-20 20:54:49,945] [INFO] [timer.py:197:stop] 0/6112, RunningAvgSamplesPerSec=5.87521868821656, CurrSamplesPerSec=5.367790798350218, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3020/5000 [10:32:30<6:18:38, 11.47s/it][2022-12-20 20:55:00,967] [INFO] [timer.py:197:stop] 0/6114, RunningAvgSamplesPerSec=5.875217580039601, CurrSamplesPerSec=5.325037000548145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3021/5000 [10:32:41<6:14:09, 11.34s/it][2022-12-20 20:55:12,016] [INFO] [timer.py:197:stop] 0/6116, RunningAvgSamplesPerSec=5.875211745587569, CurrSamplesPerSec=5.317629249658798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3022/5000 [10:32:52<6:10:37, 11.24s/it][2022-12-20 20:55:23,019] [INFO] [timer.py:197:stop] 0/6118, RunningAvgSamplesPerSec=5.875213952589403, CurrSamplesPerSec=5.31906606366517, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3023/5000 [10:33:03<6:08:05, 11.17s/it][2022-12-20 20:55:34,006] [INFO] [logging.py:68:log_dist] [Rank 0] step=3060, skipped=3, lr=[4.32e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:55:34,007] [INFO] [timer.py:197:stop] 0/6120, RunningAvgSamplesPerSec=5.875218848288103, CurrSamplesPerSec=5.33480557698934, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3024/5000 [10:33:14<6:06:23, 11.13s/it][2022-12-20 20:55:45,039] [INFO] [timer.py:197:stop] 0/6122, RunningAvgSamplesPerSec=5.875216046553836, CurrSamplesPerSec=5.323063425283123, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 60%|██████    | 3025/5000 [10:33:25<6:04:46, 11.08s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 4.3177777777777776e-06, 'epoch': 73.77}

 60%|██████    | 3025/5000 [10:33:25<6:04:46, 11.08s/it][2022-12-20 20:55:56,031] [INFO] [timer.py:197:stop] 0/6124, RunningAvgSamplesPerSec=5.875220431909395, CurrSamplesPerSec=5.313250684534988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3026/5000 [10:33:36<6:04:05, 11.07s/it][2022-12-20 20:56:07,067] [INFO] [timer.py:197:stop] 0/6126, RunningAvgSamplesPerSec=5.8752169523771345, CurrSamplesPerSec=5.308794910052335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3027/5000 [10:33:47<6:03:35, 11.06s/it][2022-12-20 20:56:18,102] [INFO] [timer.py:197:stop] 0/6128, RunningAvgSamplesPerSec=5.875214034065897, CurrSamplesPerSec=5.309694415228059, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3028/5000 [10:33:58<6:03:00, 11.04s/it][2022-12-20 20:56:29,100] [INFO] [timer.py:197:stop] 0/6130, RunningAvgSamplesPerSec=5.875217472758273, CurrSamplesPerSec=5.324754126645204, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3029/5000 [10:34:09<6:02:23, 11.03s/it][2022-12-20 20:56:40,148] [INFO] [timer.py:197:stop] 0/6132, RunningAvgSamplesPerSec=5.8752118857452365, CurrSamplesPerSec=5.282699170418777, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3030/5000 [10:34:20<6:02:23, 11.04s/it][2022-12-20 20:56:51,141] [INFO] [timer.py:197:stop] 0/6134, RunningAvgSamplesPerSec=5.87521646778979, CurrSamplesPerSec=5.332765427774333, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3031/5000 [10:34:31<6:01:54, 11.03s/it][2022-12-20 20:57:02,134] [INFO] [timer.py:197:stop] 0/6136, RunningAvgSamplesPerSec=5.875220844890712, CurrSamplesPerSec=5.347056961422694, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3032/5000 [10:34:42<6:01:13, 11.01s/it][2022-12-20 20:57:13,122] [INFO] [timer.py:197:stop] 0/6138, RunningAvgSamplesPerSec=5.8752259924068015, CurrSamplesPerSec=5.336429060612356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3033/5000 [10:34:53<6:01:02, 11.01s/it][2022-12-20 20:57:24,164] [INFO] [logging.py:68:log_dist] [Rank 0] step=3070, skipped=3, lr=[4.297777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:57:24,166] [INFO] [timer.py:197:stop] 0/6140, RunningAvgSamplesPerSec=5.8752213918464085, CurrSamplesPerSec=5.309526798116403, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3034/5000 [10:35:04<6:00:42, 11.01s/it][2022-12-20 20:57:34,237] [INFO] [timer.py:197:stop] 0/6142, RunningAvgSamplesPerSec=5.8753859191633735, CurrSamplesPerSec=6.262189267073503, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 20:57:45,224] [INFO] [timer.py:197:stop] 0/6144, RunningAvgSamplesPerSec=5.875390684367076, CurrSamplesPerSec=5.333592957312771, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3035/5000 [10:35:20<6:51:03, 12.55s/it][2022-12-20 20:57:56,230] [INFO] [timer.py:197:stop] 0/6146, RunningAvgSamplesPerSec=5.875392736458048, CurrSamplesPerSec=5.3180021819155225, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3036/5000 [10:35:31<6:35:40, 12.09s/it][2022-12-20 20:58:07,244] [INFO] [timer.py:197:stop] 0/6148, RunningAvgSamplesPerSec=5.875392875779637, CurrSamplesPerSec=5.334751294112457, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3037/5000 [10:35:42<6:24:56, 11.77s/it][2022-12-20 20:58:18,262] [INFO] [timer.py:197:stop] 0/6150, RunningAvgSamplesPerSec=5.87539258499823, CurrSamplesPerSec=5.3246867399091276, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3038/5000 [10:35:53<6:17:24, 11.54s/it][2022-12-20 20:58:29,301] [INFO] [timer.py:197:stop] 0/6152, RunningAvgSamplesPerSec=5.875388592712671, CurrSamplesPerSec=5.323922579605136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3039/5000 [10:36:04<6:12:17, 11.39s/it][2022-12-20 20:58:40,328] [INFO] [timer.py:197:stop] 0/6154, RunningAvgSamplesPerSec=5.875386891070727, CurrSamplesPerSec=5.332875608948258, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3040/5000 [10:36:15<6:08:31, 11.28s/it][2022-12-20 20:58:51,347] [INFO] [timer.py:197:stop] 0/6156, RunningAvgSamplesPerSec=5.875386699436804, CurrSamplesPerSec=5.325776544060893, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3041/5000 [10:36:26<6:05:46, 11.20s/it][2022-12-20 20:59:02,327] [INFO] [timer.py:197:stop] 0/6158, RunningAvgSamplesPerSec=5.875393301675989, CurrSamplesPerSec=5.3415337465722885, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3042/5000 [10:36:37<6:03:24, 11.14s/it][2022-12-20 20:59:13,277] [INFO] [logging.py:68:log_dist] [Rank 0] step=3080, skipped=3, lr=[4.275555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 20:59:13,278] [INFO] [timer.py:197:stop] 0/6160, RunningAvgSamplesPerSec=5.875405086218458, CurrSamplesPerSec=5.364144447640836, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3043/5000 [10:36:48<6:01:24, 11.08s/it][2022-12-20 20:59:24,296] [INFO] [timer.py:197:stop] 0/6162, RunningAvgSamplesPerSec=5.875405067480711, CurrSamplesPerSec=5.325974987969529, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3044/5000 [10:36:59<6:00:36, 11.06s/it][2022-12-20 20:59:35,294] [INFO] [timer.py:197:stop] 0/6164, RunningAvgSamplesPerSec=5.875408392859427, CurrSamplesPerSec=5.350447187676909, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3045/5000 [10:37:10<5:59:48, 11.04s/it][2022-12-20 20:59:46,293] [INFO] [timer.py:197:stop] 0/6166, RunningAvgSamplesPerSec=5.87541185261061, CurrSamplesPerSec=5.343934213471608, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3046/5000 [10:37:21<5:59:11, 11.03s/it][2022-12-20 20:59:57,320] [INFO] [timer.py:197:stop] 0/6168, RunningAvgSamplesPerSec=5.875410233910452, CurrSamplesPerSec=5.300251286394974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3047/5000 [10:37:32<5:58:59, 11.03s/it][2022-12-20 21:00:08,340] [INFO] [timer.py:197:stop] 0/6170, RunningAvgSamplesPerSec=5.875409622782899, CurrSamplesPerSec=5.304067599513908, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3048/5000 [10:37:43<5:58:42, 11.03s/it][2022-12-20 21:00:19,371] [INFO] [timer.py:197:stop] 0/6172, RunningAvgSamplesPerSec=5.875406870020345, CurrSamplesPerSec=5.292491526132839, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3049/5000 [10:37:54<5:58:34, 11.03s/it][2022-12-20 21:00:30,329] [INFO] [timer.py:197:stop] 0/6174, RunningAvgSamplesPerSec=5.87541702171069, CurrSamplesPerSec=5.343037534419127, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3050/5000 [10:38:05<5:57:42, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 4.26e-06, 'epoch': 74.39}

 61%|██████    | 3050/5000 [10:38:05<5:57:42, 11.01s/it][2022-12-20 21:00:41,295] [INFO] [timer.py:197:stop] 0/6176, RunningAvgSamplesPerSec=5.875425932188193, CurrSamplesPerSec=5.33471821602604, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3051/5000 [10:38:16<5:57:08, 10.99s/it][2022-12-20 21:00:52,346] [INFO] [timer.py:197:stop] 0/6178, RunningAvgSamplesPerSec=5.875419752058925, CurrSamplesPerSec=5.307340343606874, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3052/5000 [10:38:27<5:57:30, 11.01s/it][2022-12-20 21:01:03,380] [INFO] [logging.py:68:log_dist] [Rank 0] step=3090, skipped=3, lr=[4.253333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:01:03,382] [INFO] [timer.py:197:stop] 0/6180, RunningAvgSamplesPerSec=5.875416121444207, CurrSamplesPerSec=5.312549942060481, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3053/5000 [10:38:38<5:57:33, 11.02s/it][2022-12-20 21:01:14,397] [INFO] [timer.py:197:stop] 0/6182, RunningAvgSamplesPerSec=5.875416191920165, CurrSamplesPerSec=5.30522363089849, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3054/5000 [10:38:49<5:57:20, 11.02s/it][2022-12-20 21:01:25,423] [INFO] [timer.py:197:stop] 0/6184, RunningAvgSamplesPerSec=5.8754146101603455, CurrSamplesPerSec=5.327931688265228, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3055/5000 [10:39:01<5:57:14, 11.02s/it][2022-12-20 21:01:36,356] [INFO] [timer.py:197:stop] 0/6186, RunningAvgSamplesPerSec=5.875429429103313, CurrSamplesPerSec=5.360922059053901, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3056/5000 [10:39:11<5:56:12, 10.99s/it][2022-12-20 21:01:47,363] [INFO] [timer.py:197:stop] 0/6188, RunningAvgSamplesPerSec=5.875430814632638, CurrSamplesPerSec=5.329813632458138, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3057/5000 [10:39:22<5:56:09, 11.00s/it][2022-12-20 21:01:58,405] [INFO] [timer.py:197:stop] 0/6190, RunningAvgSamplesPerSec=5.875426277662624, CurrSamplesPerSec=5.306155279897708, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3058/5000 [10:39:34<5:56:23, 11.01s/it][2022-12-20 21:02:09,350] [INFO] [timer.py:197:stop] 0/6192, RunningAvgSamplesPerSec=5.875438891624383, CurrSamplesPerSec=5.353111865021125, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3059/5000 [10:39:44<5:55:33, 10.99s/it][2022-12-20 21:02:20,353] [INFO] [timer.py:197:stop] 0/6194, RunningAvgSamplesPerSec=5.875441272566205, CurrSamplesPerSec=5.3169366193947, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3060/5000 [10:39:55<5:55:29, 10.99s/it][2022-12-20 21:02:31,293] [INFO] [timer.py:197:stop] 0/6196, RunningAvgSamplesPerSec=5.875454789629599, CurrSamplesPerSec=5.372090313643407, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3061/5000 [10:40:06<5:54:47, 10.98s/it][2022-12-20 21:02:42,274] [INFO] [timer.py:197:stop] 0/6198, RunningAvgSamplesPerSec=5.875460861624813, CurrSamplesPerSec=5.359835376550583, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████    | 3062/5000 [10:40:17<5:54:37, 10.98s/it][2022-12-20 21:02:53,284] [INFO] [logging.py:68:log_dist] [Rank 0] step=3100, skipped=3, lr=[4.2311111111111114e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:02:53,286] [INFO] [timer.py:197:stop] 0/6200, RunningAvgSamplesPerSec=5.875461442902075, CurrSamplesPerSec=5.325920250632796, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3063/5000 [10:40:28<5:54:45, 10.99s/it][2022-12-20 21:03:04,347] [INFO] [timer.py:197:stop] 0/6202, RunningAvgSamplesPerSec=5.875453717439492, CurrSamplesPerSec=5.292645129550349, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3064/5000 [10:40:39<5:55:16, 11.01s/it][2022-12-20 21:03:15,302] [INFO] [timer.py:197:stop] 0/6204, RunningAvgSamplesPerSec=5.875464536084555, CurrSamplesPerSec=5.359400484486505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3065/5000 [10:40:50<5:54:33, 10.99s/it][2022-12-20 21:03:26,431] [INFO] [timer.py:197:stop] 0/6206, RunningAvgSamplesPerSec=5.875445052505861, CurrSamplesPerSec=5.263628772277219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3066/5000 [10:41:02<5:55:40, 11.03s/it][2022-12-20 21:03:37,439] [INFO] [timer.py:197:stop] 0/6208, RunningAvgSamplesPerSec=5.875446520257692, CurrSamplesPerSec=5.347423167340086, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3067/5000 [10:41:13<5:55:14, 11.03s/it][2022-12-20 21:03:48,485] [INFO] [timer.py:197:stop] 0/6210, RunningAvgSamplesPerSec=5.875441588749201, CurrSamplesPerSec=5.319366674725249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3068/5000 [10:41:24<5:55:14, 11.03s/it][2022-12-20 21:03:59,513] [INFO] [timer.py:197:stop] 0/6212, RunningAvgSamplesPerSec=5.875439669538957, CurrSamplesPerSec=5.314527091143623, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3069/5000 [10:41:35<5:55:00, 11.03s/it][2022-12-20 21:04:10,561] [INFO] [timer.py:197:stop] 0/6214, RunningAvgSamplesPerSec=5.875434042459262, CurrSamplesPerSec=5.305987676395846, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3070/5000 [10:41:46<5:54:59, 11.04s/it][2022-12-20 21:04:21,592] [INFO] [timer.py:197:stop] 0/6216, RunningAvgSamplesPerSec=5.875431569680698, CurrSamplesPerSec=5.296343693144335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3071/5000 [10:41:57<5:54:45, 11.03s/it][2022-12-20 21:04:32,559] [INFO] [timer.py:197:stop] 0/6218, RunningAvgSamplesPerSec=5.875440040521494, CurrSamplesPerSec=5.350358247269186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3072/5000 [10:42:08<5:53:55, 11.01s/it][2022-12-20 21:04:43,603] [INFO] [logging.py:68:log_dist] [Rank 0] step=3110, skipped=3, lr=[4.208888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:04:43,605] [INFO] [timer.py:197:stop] 0/6220, RunningAvgSamplesPerSec=5.8754348591508885, CurrSamplesPerSec=5.310994332152723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3073/5000 [10:42:19<5:54:02, 11.02s/it][2022-12-20 21:04:54,613] [INFO] [timer.py:197:stop] 0/6222, RunningAvgSamplesPerSec=5.875436265342186, CurrSamplesPerSec=5.331052484428889, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 61%|██████▏   | 3074/5000 [10:42:30<5:53:42, 11.02s/it][2022-12-20 21:05:05,621] [INFO] [timer.py:197:stop] 0/6224, RunningAvgSamplesPerSec=5.875437567245927, CurrSamplesPerSec=5.335063647806382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3075/5000 [10:42:41<5:53:25, 11.02s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 4.204444444444445e-06, 'epoch': 74.99}

 62%|██████▏   | 3075/5000 [10:42:41<5:53:25, 11.02s/it][2022-12-20 21:05:15,656] [INFO] [timer.py:197:stop] 0/6226, RunningAvgSamplesPerSec=5.875606319456273, CurrSamplesPerSec=5.350819617755565, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3076/5000 [10:42:56<6:31:21, 12.20s/it][2022-12-20 21:05:26,634] [INFO] [timer.py:197:stop] 0/6228, RunningAvgSamplesPerSec=5.875612595127981, CurrSamplesPerSec=5.3595562841601625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3077/5000 [10:43:07<6:19:35, 11.84s/it][2022-12-20 21:05:37,668] [INFO] [timer.py:197:stop] 0/6230, RunningAvgSamplesPerSec=5.875609332267287, CurrSamplesPerSec=5.330502847450952, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3078/5000 [10:43:18<6:11:25, 11.59s/it][2022-12-20 21:05:48,719] [INFO] [timer.py:197:stop] 0/6232, RunningAvgSamplesPerSec=5.875603210604129, CurrSamplesPerSec=5.298652665859183, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3079/5000 [10:43:29<6:05:40, 11.42s/it][2022-12-20 21:05:59,733] [INFO] [timer.py:197:stop] 0/6234, RunningAvgSamplesPerSec=5.875603403086806, CurrSamplesPerSec=5.300252123621727, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3080/5000 [10:43:40<6:02:03, 11.31s/it][2022-12-20 21:06:10,772] [INFO] [timer.py:197:stop] 0/6236, RunningAvgSamplesPerSec=5.875599611061532, CurrSamplesPerSec=5.324005152416805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3081/5000 [10:43:51<5:59:16, 11.23s/it][2022-12-20 21:06:21,765] [INFO] [timer.py:197:stop] 0/6238, RunningAvgSamplesPerSec=5.875603325306536, CurrSamplesPerSec=5.369089466297792, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3082/5000 [10:44:02<5:56:22, 11.15s/it][2022-12-20 21:06:32,769] [INFO] [logging.py:68:log_dist] [Rank 0] step=3120, skipped=3, lr=[4.1866666666666675e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:06:32,771] [INFO] [timer.py:197:stop] 0/6240, RunningAvgSamplesPerSec=5.87560490750612, CurrSamplesPerSec=5.3187342925868535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3083/5000 [10:44:13<5:54:50, 11.11s/it][2022-12-20 21:06:43,815] [INFO] [timer.py:197:stop] 0/6242, RunningAvgSamplesPerSec=5.875599870660265, CurrSamplesPerSec=5.2872399431228, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3084/5000 [10:44:24<5:54:26, 11.10s/it][2022-12-20 21:06:54,811] [INFO] [timer.py:197:stop] 0/6244, RunningAvgSamplesPerSec=5.875603062950178, CurrSamplesPerSec=5.364987106544963, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3085/5000 [10:44:35<5:53:14, 11.07s/it][2022-12-20 21:07:05,790] [INFO] [timer.py:197:stop] 0/6246, RunningAvgSamplesPerSec=5.875609273493416, CurrSamplesPerSec=5.377592688016897, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3086/5000 [10:44:46<5:52:16, 11.04s/it][2022-12-20 21:07:16,859] [INFO] [timer.py:197:stop] 0/6248, RunningAvgSamplesPerSec=5.875599941935591, CurrSamplesPerSec=5.303341197073117, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3087/5000 [10:44:57<5:52:17, 11.05s/it][2022-12-20 21:07:27,918] [INFO] [timer.py:197:stop] 0/6250, RunningAvgSamplesPerSec=5.875592298765122, CurrSamplesPerSec=5.307945249013958, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3088/5000 [10:45:08<5:52:05, 11.05s/it][2022-12-20 21:07:38,948] [INFO] [timer.py:197:stop] 0/6252, RunningAvgSamplesPerSec=5.875589689367221, CurrSamplesPerSec=5.322781394283794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3089/5000 [10:45:19<5:51:51, 11.05s/it][2022-12-20 21:07:49,961] [INFO] [timer.py:197:stop] 0/6254, RunningAvgSamplesPerSec=5.875590040001721, CurrSamplesPerSec=5.350465317379412, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3090/5000 [10:45:30<5:50:47, 11.02s/it][2022-12-20 21:08:00,925] [INFO] [timer.py:197:stop] 0/6256, RunningAvgSamplesPerSec=5.875599070689484, CurrSamplesPerSec=5.3423603848873205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3091/5000 [10:45:41<5:50:26, 11.01s/it][2022-12-20 21:08:11,925] [INFO] [timer.py:197:stop] 0/6258, RunningAvgSamplesPerSec=5.875601826769716, CurrSamplesPerSec=5.344804162694101, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3092/5000 [10:45:52<5:49:28, 10.99s/it][2022-12-20 21:08:22,850] [INFO] [logging.py:68:log_dist] [Rank 0] step=3130, skipped=3, lr=[4.164444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:08:22,851] [INFO] [timer.py:197:stop] 0/6260, RunningAvgSamplesPerSec=5.875617276772755, CurrSamplesPerSec=5.349955599639249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3093/5000 [10:46:03<5:49:26, 10.99s/it][2022-12-20 21:08:33,882] [INFO] [timer.py:197:stop] 0/6262, RunningAvgSamplesPerSec=5.875614709231481, CurrSamplesPerSec=5.328375872267355, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3094/5000 [10:46:14<5:49:07, 10.99s/it][2022-12-20 21:08:44,859] [INFO] [timer.py:197:stop] 0/6264, RunningAvgSamplesPerSec=5.875621276986196, CurrSamplesPerSec=5.331503542419439, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3095/5000 [10:46:25<5:49:00, 10.99s/it][2022-12-20 21:08:55,869] [INFO] [timer.py:197:stop] 0/6266, RunningAvgSamplesPerSec=5.875622204226143, CurrSamplesPerSec=5.320642225310853, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3096/5000 [10:46:36<5:49:25, 11.01s/it][2022-12-20 21:09:06,914] [INFO] [timer.py:197:stop] 0/6268, RunningAvgSamplesPerSec=5.875616969970707, CurrSamplesPerSec=5.328927398031122, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3097/5000 [10:46:47<5:49:18, 11.01s/it][2022-12-20 21:09:17,961] [INFO] [timer.py:197:stop] 0/6270, RunningAvgSamplesPerSec=5.87561161308371, CurrSamplesPerSec=5.304475318754092, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3098/5000 [10:46:58<5:49:26, 11.02s/it][2022-12-20 21:09:28,987] [INFO] [timer.py:197:stop] 0/6272, RunningAvgSamplesPerSec=5.8756098384770015, CurrSamplesPerSec=5.323357943142148, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3099/5000 [10:47:09<5:49:15, 11.02s/it][2022-12-20 21:09:40,007] [INFO] [timer.py:197:stop] 0/6274, RunningAvgSamplesPerSec=5.875608993920647, CurrSamplesPerSec=5.325873122475926, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3100/5000 [10:47:20<5:49:11, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 4.148888888888889e-06, 'epoch': 75.6}

 62%|██████▏   | 3100/5000 [10:47:20<5:49:11, 11.03s/it][2022-12-20 21:09:51,055] [INFO] [timer.py:197:stop] 0/6276, RunningAvgSamplesPerSec=5.875603524674878, CurrSamplesPerSec=5.31501155822723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3101/5000 [10:47:31<5:48:47, 11.02s/it][2022-12-20 21:10:02,074] [INFO] [timer.py:197:stop] 0/6278, RunningAvgSamplesPerSec=5.8756028379582075, CurrSamplesPerSec=5.301434761111063, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3102/5000 [10:47:42<5:48:54, 11.03s/it][2022-12-20 21:10:13,087] [INFO] [logging.py:68:log_dist] [Rank 0] step=3140, skipped=3, lr=[4.142222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:10:13,088] [INFO] [timer.py:197:stop] 0/6280, RunningAvgSamplesPerSec=5.8756031157203745, CurrSamplesPerSec=5.334563220970156, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3103/5000 [10:47:53<5:48:47, 11.03s/it][2022-12-20 21:10:24,103] [INFO] [timer.py:197:stop] 0/6282, RunningAvgSamplesPerSec=5.875603600510592, CurrSamplesPerSec=5.356465739130754, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3104/5000 [10:48:04<5:48:13, 11.02s/it][2022-12-20 21:10:35,094] [INFO] [timer.py:197:stop] 0/6284, RunningAvgSamplesPerSec=5.875607888589488, CurrSamplesPerSec=5.355285561212265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3105/5000 [10:48:15<5:47:48, 11.01s/it][2022-12-20 21:10:46,136] [INFO] [timer.py:197:stop] 0/6286, RunningAvgSamplesPerSec=5.87560332639956, CurrSamplesPerSec=5.313932256075216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3106/5000 [10:48:26<5:48:10, 11.03s/it][2022-12-20 21:10:57,170] [INFO] [timer.py:197:stop] 0/6288, RunningAvgSamplesPerSec=5.875600017426598, CurrSamplesPerSec=5.346167752136601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3107/5000 [10:48:37<5:47:30, 11.01s/it][2022-12-20 21:11:08,187] [INFO] [timer.py:197:stop] 0/6290, RunningAvgSamplesPerSec=5.875599787024507, CurrSamplesPerSec=5.313245846843472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3108/5000 [10:48:48<5:47:10, 11.01s/it][2022-12-20 21:11:19,168] [INFO] [timer.py:197:stop] 0/6292, RunningAvgSamplesPerSec=5.875605681574856, CurrSamplesPerSec=5.32825826205645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3109/5000 [10:48:59<5:46:54, 11.01s/it][2022-12-20 21:11:30,165] [INFO] [timer.py:197:stop] 0/6294, RunningAvgSamplesPerSec=5.875608850498775, CurrSamplesPerSec=5.3323327988197, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3110/5000 [10:49:10<5:46:38, 11.00s/it][2022-12-20 21:11:41,165] [INFO] [timer.py:197:stop] 0/6296, RunningAvgSamplesPerSec=5.87561154462533, CurrSamplesPerSec=5.330709689052558, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3111/5000 [10:49:21<5:46:34, 11.01s/it][2022-12-20 21:11:52,200] [INFO] [timer.py:197:stop] 0/6298, RunningAvgSamplesPerSec=5.875608168060725, CurrSamplesPerSec=5.3140008436346795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3112/5000 [10:49:32<5:46:15, 11.00s/it][2022-12-20 21:12:03,195] [INFO] [logging.py:68:log_dist] [Rank 0] step=3150, skipped=3, lr=[4.12e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:12:03,197] [INFO] [timer.py:197:stop] 0/6300, RunningAvgSamplesPerSec=5.875611124523405, CurrSamplesPerSec=5.31139344838698, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3113/5000 [10:49:43<5:45:59, 11.00s/it][2022-12-20 21:12:14,166] [INFO] [timer.py:197:stop] 0/6302, RunningAvgSamplesPerSec=5.8756187493680985, CurrSamplesPerSec=5.334053348669427, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3114/5000 [10:49:54<5:45:44, 11.00s/it][2022-12-20 21:12:25,140] [INFO] [timer.py:197:stop] 0/6304, RunningAvgSamplesPerSec=5.875625798594866, CurrSamplesPerSec=5.352168564182508, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3115/5000 [10:50:05<5:45:30, 11.00s/it][2022-12-20 21:12:36,168] [INFO] [timer.py:197:stop] 0/6306, RunningAvgSamplesPerSec=5.875623428158685, CurrSamplesPerSec=5.321727209244086, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3116/5000 [10:50:16<5:45:31, 11.00s/it][2022-12-20 21:12:46,250] [INFO] [timer.py:197:stop] 0/6308, RunningAvgSamplesPerSec=5.87578141331627, CurrSamplesPerSec=6.2929277584795145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 21:12:57,254] [INFO] [timer.py:197:stop] 0/6310, RunningAvgSamplesPerSec=5.875782690699931, CurrSamplesPerSec=5.3575266760207425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3117/5000 [10:50:32<6:33:43, 12.55s/it][2022-12-20 21:13:08,247] [INFO] [timer.py:197:stop] 0/6312, RunningAvgSamplesPerSec=5.875786610263724, CurrSamplesPerSec=5.312993878394825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3118/5000 [10:50:43<6:18:54, 12.08s/it][2022-12-20 21:13:19,263] [INFO] [timer.py:197:stop] 0/6314, RunningAvgSamplesPerSec=5.875786367770704, CurrSamplesPerSec=5.310380327315196, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3119/5000 [10:50:54<6:08:42, 11.76s/it][2022-12-20 21:13:30,277] [INFO] [timer.py:197:stop] 0/6316, RunningAvgSamplesPerSec=5.875786689064704, CurrSamplesPerSec=5.345114501761743, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3120/5000 [10:51:05<6:01:29, 11.54s/it][2022-12-20 21:13:41,256] [INFO] [timer.py:197:stop] 0/6318, RunningAvgSamplesPerSec=5.875792878768084, CurrSamplesPerSec=5.338806889715256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3121/5000 [10:51:16<5:56:02, 11.37s/it][2022-12-20 21:13:52,300] [INFO] [logging.py:68:log_dist] [Rank 0] step=3160, skipped=3, lr=[4.097777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:13:52,301] [INFO] [timer.py:197:stop] 0/6320, RunningAvgSamplesPerSec=5.875787802305916, CurrSamplesPerSec=5.31391479388119, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3122/5000 [10:51:27<5:52:48, 11.27s/it][2022-12-20 21:14:03,306] [INFO] [timer.py:197:stop] 0/6322, RunningAvgSamplesPerSec=5.875789882997917, CurrSamplesPerSec=5.3473255927406775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3123/5000 [10:51:38<5:50:07, 11.19s/it][2022-12-20 21:14:14,293] [INFO] [timer.py:197:stop] 0/6324, RunningAvgSamplesPerSec=5.875794807417099, CurrSamplesPerSec=5.358853545267827, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▏   | 3124/5000 [10:51:49<5:48:00, 11.13s/it][2022-12-20 21:14:25,308] [INFO] [timer.py:197:stop] 0/6326, RunningAvgSamplesPerSec=5.875794975849509, CurrSamplesPerSec=5.306514016341192, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 62%|██████▎   | 3125/5000 [10:52:00<5:46:44, 11.10s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 4.091111111111111e-06, 'epoch': 76.22}

 62%|██████▎   | 3125/5000 [10:52:00<5:46:44, 11.10s/it][2022-12-20 21:14:36,348] [INFO] [timer.py:197:stop] 0/6328, RunningAvgSamplesPerSec=5.875790888602485, CurrSamplesPerSec=5.310986136081386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3126/5000 [10:52:11<5:46:02, 11.08s/it][2022-12-20 21:14:47,333] [INFO] [timer.py:197:stop] 0/6330, RunningAvgSamplesPerSec=5.875796103722073, CurrSamplesPerSec=5.34866510449421, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3127/5000 [10:52:22<5:44:58, 11.05s/it][2022-12-20 21:14:58,241] [INFO] [timer.py:197:stop] 0/6332, RunningAvgSamplesPerSec=5.875814304073243, CurrSamplesPerSec=5.385724130531104, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3128/5000 [10:52:33<5:43:27, 11.01s/it][2022-12-20 21:15:09,263] [INFO] [timer.py:197:stop] 0/6334, RunningAvgSamplesPerSec=5.875813232499461, CurrSamplesPerSec=5.319812172809861, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3129/5000 [10:52:44<5:43:23, 11.01s/it][2022-12-20 21:15:20,262] [INFO] [timer.py:197:stop] 0/6336, RunningAvgSamplesPerSec=5.875815891111635, CurrSamplesPerSec=5.347012866711517, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3130/5000 [10:52:55<5:43:05, 11.01s/it][2022-12-20 21:15:31,298] [INFO] [timer.py:197:stop] 0/6338, RunningAvgSamplesPerSec=5.875812305076677, CurrSamplesPerSec=5.304510328932238, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3131/5000 [10:53:06<5:43:09, 11.02s/it][2022-12-20 21:15:42,338] [INFO] [logging.py:68:log_dist] [Rank 0] step=3170, skipped=3, lr=[4.075555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:15:42,339] [INFO] [timer.py:197:stop] 0/6340, RunningAvgSamplesPerSec=5.875807642541231, CurrSamplesPerSec=5.325972663194898, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3132/5000 [10:53:17<5:43:12, 11.02s/it][2022-12-20 21:15:53,287] [INFO] [timer.py:197:stop] 0/6342, RunningAvgSamplesPerSec=5.875819078930652, CurrSamplesPerSec=5.3852042159215365, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3133/5000 [10:53:28<5:42:19, 11.00s/it][2022-12-20 21:16:04,273] [INFO] [timer.py:197:stop] 0/6344, RunningAvgSamplesPerSec=5.875824032444822, CurrSamplesPerSec=5.334759563698155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3134/5000 [10:53:39<5:41:59, 11.00s/it][2022-12-20 21:16:15,294] [INFO] [timer.py:197:stop] 0/6346, RunningAvgSamplesPerSec=5.875823489378198, CurrSamplesPerSec=5.328289356337546, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3135/5000 [10:53:50<5:42:02, 11.00s/it][2022-12-20 21:16:26,282] [INFO] [timer.py:197:stop] 0/6348, RunningAvgSamplesPerSec=5.8758283019878395, CurrSamplesPerSec=5.312495059774893, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3136/5000 [10:54:01<5:41:42, 11.00s/it][2022-12-20 21:16:37,324] [INFO] [timer.py:197:stop] 0/6350, RunningAvgSamplesPerSec=5.8758240164387505, CurrSamplesPerSec=5.317266902317515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3137/5000 [10:54:12<5:41:55, 11.01s/it][2022-12-20 21:16:48,321] [INFO] [timer.py:197:stop] 0/6352, RunningAvgSamplesPerSec=5.875827262274412, CurrSamplesPerSec=5.310300487699732, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3138/5000 [10:54:23<5:41:35, 11.01s/it][2022-12-20 21:16:59,316] [INFO] [timer.py:197:stop] 0/6354, RunningAvgSamplesPerSec=5.87583074297457, CurrSamplesPerSec=5.360196056622872, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3139/5000 [10:54:34<5:41:17, 11.00s/it][2022-12-20 21:17:10,309] [INFO] [timer.py:197:stop] 0/6356, RunningAvgSamplesPerSec=5.875834519934735, CurrSamplesPerSec=5.329756276468057, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3140/5000 [10:54:45<5:41:01, 11.00s/it][2022-12-20 21:17:21,269] [INFO] [timer.py:197:stop] 0/6358, RunningAvgSamplesPerSec=5.875844222828617, CurrSamplesPerSec=5.3575777877818815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3141/5000 [10:54:56<5:40:27, 10.99s/it][2022-12-20 21:17:32,219] [INFO] [logging.py:68:log_dist] [Rank 0] step=3180, skipped=3, lr=[4.053333333333333e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:17:32,221] [INFO] [timer.py:197:stop] 0/6360, RunningAvgSamplesPerSec=5.875855027408668, CurrSamplesPerSec=5.326508894511631, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3142/5000 [10:55:07<5:39:55, 10.98s/it][2022-12-20 21:17:43,214] [INFO] [timer.py:197:stop] 0/6362, RunningAvgSamplesPerSec=5.875858775970194, CurrSamplesPerSec=5.367408274930487, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3143/5000 [10:55:18<5:39:53, 10.98s/it][2022-12-20 21:17:54,223] [INFO] [timer.py:197:stop] 0/6364, RunningAvgSamplesPerSec=5.875859907437065, CurrSamplesPerSec=5.348037244677403, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3144/5000 [10:55:29<5:39:57, 10.99s/it][2022-12-20 21:18:05,177] [INFO] [timer.py:197:stop] 0/6366, RunningAvgSamplesPerSec=5.875870309526034, CurrSamplesPerSec=5.346936181989491, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3145/5000 [10:55:40<5:39:26, 10.98s/it][2022-12-20 21:18:16,166] [INFO] [timer.py:197:stop] 0/6368, RunningAvgSamplesPerSec=5.8758746784421065, CurrSamplesPerSec=5.33175981123773, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3146/5000 [10:55:51<5:39:21, 10.98s/it][2022-12-20 21:18:26,508] [INFO] [stage_1_and_2.py:1765:step] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072.0, reducing to 131072.0
[2022-12-20 21:18:26,510] [INFO] [timer.py:197:stop] 0/6370, RunningAvgSamplesPerSec=5.875988284750518, CurrSamplesPerSec=5.9390995257622645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3147/5000 [10:56:02<5:33:14, 10.79s/it][2022-12-20 21:18:36,903] [INFO] [stage_1_and_2.py:1765:step] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2022-12-20 21:18:36,905] [INFO] [timer.py:197:stop] 0/6372, RunningAvgSamplesPerSec=5.876093041006348, CurrSamplesPerSec=5.932779524200002, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3148/5000 [10:56:12<5:29:24, 10.67s/it][2022-12-20 21:18:47,870] [INFO] [timer.py:197:stop] 0/6374, RunningAvgSamplesPerSec=5.876101254158571, CurrSamplesPerSec=5.355648834751575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3149/5000 [10:56:23<5:31:56, 10.76s/it][2022-12-20 21:18:58,946] [INFO] [timer.py:197:stop] 0/6376, RunningAvgSamplesPerSec=5.876090683937146, CurrSamplesPerSec=5.291397777015493, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3150/5000 [10:56:34<5:34:41, 10.85s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 4.04e-06, 'epoch': 76.82}

 63%|██████▎   | 3150/5000 [10:56:34<5:34:41, 10.85s/it][2022-12-20 21:19:09,901] [INFO] [timer.py:197:stop] 0/6378, RunningAvgSamplesPerSec=5.8761012120645475, CurrSamplesPerSec=5.351963681953357, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3151/5000 [10:56:45<5:35:25, 10.88s/it][2022-12-20 21:19:20,947] [INFO] [logging.py:68:log_dist] [Rank 0] step=3190, skipped=5, lr=[4.035555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:19:20,949] [INFO] [timer.py:197:stop] 0/6380, RunningAvgSamplesPerSec=5.8760955506500965, CurrSamplesPerSec=5.3080676321433415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3152/5000 [10:56:56<5:36:45, 10.93s/it][2022-12-20 21:19:31,931] [INFO] [timer.py:197:stop] 0/6382, RunningAvgSamplesPerSec=5.87610101982182, CurrSamplesPerSec=5.318680757803182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3153/5000 [10:57:07<5:37:01, 10.95s/it][2022-12-20 21:19:42,935] [INFO] [timer.py:197:stop] 0/6384, RunningAvgSamplesPerSec=5.87610297034574, CurrSamplesPerSec=5.306512547730559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3154/5000 [10:57:18<5:37:21, 10.96s/it][2022-12-20 21:19:53,985] [INFO] [timer.py:197:stop] 0/6386, RunningAvgSamplesPerSec=5.876097098631627, CurrSamplesPerSec=5.309603883869695, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3155/5000 [10:57:29<5:37:57, 10.99s/it][2022-12-20 21:20:04,988] [INFO] [timer.py:197:stop] 0/6388, RunningAvgSamplesPerSec=5.876099063138022, CurrSamplesPerSec=5.341944482398727, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3156/5000 [10:57:40<5:37:53, 10.99s/it][2022-12-20 21:20:15,981] [INFO] [timer.py:197:stop] 0/6390, RunningAvgSamplesPerSec=5.87610258844795, CurrSamplesPerSec=5.354727711755498, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3157/5000 [10:57:51<5:37:42, 10.99s/it][2022-12-20 21:20:26,081] [INFO] [timer.py:197:stop] 0/6392, RunningAvgSamplesPerSec=5.876255369318709, CurrSamplesPerSec=5.332155699675131, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3158/5000 [10:58:06<6:15:00, 12.22s/it][2022-12-20 21:20:37,126] [INFO] [timer.py:197:stop] 0/6394, RunningAvgSamplesPerSec=5.876250159910572, CurrSamplesPerSec=5.319182003597226, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3159/5000 [10:58:17<6:03:42, 11.85s/it][2022-12-20 21:20:48,105] [INFO] [timer.py:197:stop] 0/6396, RunningAvgSamplesPerSec=5.876255961658893, CurrSamplesPerSec=5.346189685957223, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3160/5000 [10:58:28<5:55:47, 11.60s/it][2022-12-20 21:20:59,198] [INFO] [timer.py:197:stop] 0/6398, RunningAvgSamplesPerSec=5.876242603432418, CurrSamplesPerSec=5.276474861999291, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3161/5000 [10:58:39<5:50:34, 11.44s/it][2022-12-20 21:21:10,212] [INFO] [logging.py:68:log_dist] [Rank 0] step=3200, skipped=5, lr=[4.013333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:21:10,213] [INFO] [timer.py:197:stop] 0/6400, RunningAvgSamplesPerSec=5.8762422826888905, CurrSamplesPerSec=5.312503470779569, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3162/5000 [10:58:50<5:46:30, 11.31s/it][2022-12-20 21:21:21,184] [INFO] [timer.py:197:stop] 0/6402, RunningAvgSamplesPerSec=5.876249427693909, CurrSamplesPerSec=5.352740823575111, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3163/5000 [10:59:01<5:43:10, 11.21s/it][2022-12-20 21:21:32,197] [INFO] [timer.py:197:stop] 0/6404, RunningAvgSamplesPerSec=5.876249531632792, CurrSamplesPerSec=5.313683378052247, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3164/5000 [10:59:12<5:41:30, 11.16s/it][2022-12-20 21:21:43,263] [INFO] [timer.py:197:stop] 0/6406, RunningAvgSamplesPerSec=5.876240704356938, CurrSamplesPerSec=5.296535978420769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3165/5000 [10:59:23<5:40:18, 11.13s/it][2022-12-20 21:21:54,208] [INFO] [timer.py:197:stop] 0/6408, RunningAvgSamplesPerSec=5.876252614638468, CurrSamplesPerSec=5.392621080394686, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3166/5000 [10:59:34<5:38:27, 11.07s/it][2022-12-20 21:22:05,252] [INFO] [timer.py:197:stop] 0/6410, RunningAvgSamplesPerSec=5.8762475570118164, CurrSamplesPerSec=5.304394189159511, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3167/5000 [10:59:45<5:38:17, 11.07s/it][2022-12-20 21:22:16,327] [INFO] [timer.py:197:stop] 0/6412, RunningAvgSamplesPerSec=5.876237125687304, CurrSamplesPerSec=5.30281234368005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3168/5000 [10:59:56<5:37:30, 11.05s/it][2022-12-20 21:22:27,323] [INFO] [timer.py:197:stop] 0/6414, RunningAvgSamplesPerSec=5.876240386141141, CurrSamplesPerSec=5.315047970486134, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3169/5000 [11:00:07<5:36:57, 11.04s/it][2022-12-20 21:22:38,332] [INFO] [timer.py:197:stop] 0/6416, RunningAvgSamplesPerSec=5.876241208720822, CurrSamplesPerSec=5.319394924635874, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3170/5000 [11:00:18<5:36:39, 11.04s/it][2022-12-20 21:22:49,363] [INFO] [timer.py:197:stop] 0/6418, RunningAvgSamplesPerSec=5.876238226311194, CurrSamplesPerSec=5.317464923233397, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3171/5000 [11:00:29<5:36:34, 11.04s/it][2022-12-20 21:23:00,437] [INFO] [logging.py:68:log_dist] [Rank 0] step=3210, skipped=5, lr=[3.991111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:23:00,438] [INFO] [timer.py:197:stop] 0/6420, RunningAvgSamplesPerSec=5.876228189137, CurrSamplesPerSec=5.293966570610825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3172/5000 [11:00:40<5:36:41, 11.05s/it][2022-12-20 21:23:11,481] [INFO] [timer.py:197:stop] 0/6422, RunningAvgSamplesPerSec=5.876223259937705, CurrSamplesPerSec=5.320739672240535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3173/5000 [11:00:52<5:36:28, 11.05s/it][2022-12-20 21:23:22,499] [INFO] [timer.py:197:stop] 0/6424, RunningAvgSamplesPerSec=5.876222503710531, CurrSamplesPerSec=5.347268285301791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 63%|██████▎   | 3174/5000 [11:01:03<5:35:31, 11.03s/it][2022-12-20 21:23:33,470] [INFO] [timer.py:197:stop] 0/6426, RunningAvgSamplesPerSec=5.876229675108204, CurrSamplesPerSec=5.343587844823914, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3175/5000 [11:01:14<5:35:13, 11.02s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.984444444444444e-06, 'epoch': 77.43}

 64%|██████▎   | 3175/5000 [11:01:14<5:35:13, 11.02s/it][2022-12-20 21:23:44,447] [INFO] [timer.py:197:stop] 0/6428, RunningAvgSamplesPerSec=5.876236012038305, CurrSamplesPerSec=5.376032351268899, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3176/5000 [11:01:25<5:34:42, 11.01s/it][2022-12-20 21:23:55,485] [INFO] [timer.py:197:stop] 0/6430, RunningAvgSamplesPerSec=5.876231974081519, CurrSamplesPerSec=5.327617208471063, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3177/5000 [11:01:36<5:34:20, 11.00s/it][2022-12-20 21:24:06,489] [INFO] [timer.py:197:stop] 0/6432, RunningAvgSamplesPerSec=5.87623357983966, CurrSamplesPerSec=5.315034710472819, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3178/5000 [11:01:47<5:34:48, 11.03s/it][2022-12-20 21:24:17,573] [INFO] [timer.py:197:stop] 0/6434, RunningAvgSamplesPerSec=5.876221957645018, CurrSamplesPerSec=5.307427229935187, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3179/5000 [11:01:58<5:34:50, 11.03s/it][2022-12-20 21:24:28,622] [INFO] [timer.py:197:stop] 0/6436, RunningAvgSamplesPerSec=5.8762160861004995, CurrSamplesPerSec=5.307512230183054, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3180/5000 [11:02:09<5:34:16, 11.02s/it][2022-12-20 21:24:39,589] [INFO] [timer.py:197:stop] 0/6438, RunningAvgSamplesPerSec=5.8762242507962545, CurrSamplesPerSec=5.32945279705313, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3181/5000 [11:02:20<5:33:21, 11.00s/it][2022-12-20 21:24:50,520] [INFO] [logging.py:68:log_dist] [Rank 0] step=3220, skipped=5, lr=[3.96888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:24:50,522] [INFO] [timer.py:197:stop] 0/6440, RunningAvgSamplesPerSec=5.876237790325284, CurrSamplesPerSec=5.334250501867993, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3182/5000 [11:02:31<5:32:53, 10.99s/it][2022-12-20 21:25:01,511] [INFO] [timer.py:197:stop] 0/6442, RunningAvgSamplesPerSec=5.876244336048273, CurrSamplesPerSec=5.324572249741662, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3183/5000 [11:02:42<5:33:03, 11.00s/it][2022-12-20 21:25:12,588] [INFO] [timer.py:197:stop] 0/6444, RunningAvgSamplesPerSec=5.8762339286776815, CurrSamplesPerSec=5.267048345117353, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3184/5000 [11:02:53<5:33:04, 11.00s/it][2022-12-20 21:25:23,555] [INFO] [timer.py:197:stop] 0/6446, RunningAvgSamplesPerSec=5.876241751250619, CurrSamplesPerSec=5.312695248912615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3185/5000 [11:03:04<5:33:03, 11.01s/it][2022-12-20 21:25:34,568] [INFO] [timer.py:197:stop] 0/6448, RunningAvgSamplesPerSec=5.876241997190707, CurrSamplesPerSec=5.321844320294171, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3186/5000 [11:03:15<5:33:00, 11.01s/it][2022-12-20 21:25:45,599] [INFO] [timer.py:197:stop] 0/6450, RunningAvgSamplesPerSec=5.876238983452693, CurrSamplesPerSec=5.315616318956953, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▎   | 3187/5000 [11:03:26<5:32:33, 11.01s/it][2022-12-20 21:25:56,593] [INFO] [timer.py:197:stop] 0/6452, RunningAvgSamplesPerSec=5.876242463045907, CurrSamplesPerSec=5.30939321611673, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3188/5000 [11:03:37<5:32:34, 11.01s/it][2022-12-20 21:26:07,596] [INFO] [timer.py:197:stop] 0/6454, RunningAvgSamplesPerSec=5.876244648670828, CurrSamplesPerSec=5.332781742776376, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3189/5000 [11:03:48<5:32:19, 11.01s/it][2022-12-20 21:26:18,583] [INFO] [timer.py:197:stop] 0/6456, RunningAvgSamplesPerSec=5.876249417688414, CurrSamplesPerSec=5.349450882843727, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3190/5000 [11:03:59<5:32:13, 11.01s/it][2022-12-20 21:26:29,658] [INFO] [timer.py:197:stop] 0/6458, RunningAvgSamplesPerSec=5.87623911382305, CurrSamplesPerSec=5.297866682466066, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3191/5000 [11:04:10<5:32:12, 11.02s/it][2022-12-20 21:26:40,663] [INFO] [logging.py:68:log_dist] [Rank 0] step=3230, skipped=5, lr=[3.946666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:26:40,665] [INFO] [timer.py:197:stop] 0/6460, RunningAvgSamplesPerSec=5.876240351516524, CurrSamplesPerSec=5.319230910684234, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3192/5000 [11:04:21<5:32:02, 11.02s/it][2022-12-20 21:26:51,718] [INFO] [timer.py:197:stop] 0/6462, RunningAvgSamplesPerSec=5.876233874451713, CurrSamplesPerSec=5.29088590253055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3193/5000 [11:04:32<5:32:08, 11.03s/it][2022-12-20 21:27:02,702] [INFO] [timer.py:197:stop] 0/6464, RunningAvgSamplesPerSec=5.876239042473475, CurrSamplesPerSec=5.35012087404007, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3194/5000 [11:04:43<5:31:36, 11.02s/it][2022-12-20 21:27:13,710] [INFO] [timer.py:197:stop] 0/6466, RunningAvgSamplesPerSec=5.876240278084935, CurrSamplesPerSec=5.335167985906205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3195/5000 [11:04:54<5:30:59, 11.00s/it][2022-12-20 21:27:24,684] [INFO] [timer.py:197:stop] 0/6468, RunningAvgSamplesPerSec=5.876247041142377, CurrSamplesPerSec=5.329727916397701, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3196/5000 [11:05:05<5:31:02, 11.01s/it][2022-12-20 21:27:35,722] [INFO] [timer.py:197:stop] 0/6470, RunningAvgSamplesPerSec=5.876243094220484, CurrSamplesPerSec=5.321609470357277, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3197/5000 [11:05:16<5:30:37, 11.00s/it][2022-12-20 21:27:46,705] [INFO] [timer.py:197:stop] 0/6472, RunningAvgSamplesPerSec=5.876248575700225, CurrSamplesPerSec=5.323194106963349, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3198/5000 [11:05:27<5:30:28, 11.00s/it][2022-12-20 21:27:56,802] [INFO] [timer.py:197:stop] 0/6474, RunningAvgSamplesPerSec=5.876399826978776, CurrSamplesPerSec=6.25827132075074, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 21:28:07,762] [INFO] [timer.py:197:stop] 0/6476, RunningAvgSamplesPerSec=5.876408493150059, CurrSamplesPerSec=5.31331126159194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3199/5000 [11:05:43<6:16:25, 12.54s/it][2022-12-20 21:28:18,829] [INFO] [timer.py:197:stop] 0/6478, RunningAvgSamplesPerSec=5.876400711603962, CurrSamplesPerSec=5.292020336495706, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3200/5000 [11:05:54<6:02:57, 12.10s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.926666666666667e-06, 'epoch': 78.05}

 64%|██████▍   | 3200/5000 [11:05:54<6:02:57, 12.10s/it][2022-12-20 21:28:29,867] [INFO] [logging.py:68:log_dist] [Rank 0] step=3240, skipped=5, lr=[3.924444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:28:29,868] [INFO] [timer.py:197:stop] 0/6480, RunningAvgSamplesPerSec=5.876396856253623, CurrSamplesPerSec=5.320810123109742, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3201/5000 [11:06:05<5:53:13, 11.78s/it][2022-12-20 21:28:40,935] [INFO] [timer.py:197:stop] 0/6482, RunningAvgSamplesPerSec=5.876388225205631, CurrSamplesPerSec=5.2810870142081034, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3202/5000 [11:06:16<5:46:36, 11.57s/it][2022-12-20 21:28:51,981] [INFO] [timer.py:197:stop] 0/6484, RunningAvgSamplesPerSec=5.876382994866593, CurrSamplesPerSec=5.320551952985218, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3203/5000 [11:06:27<5:41:44, 11.41s/it][2022-12-20 21:29:02,983] [INFO] [timer.py:197:stop] 0/6486, RunningAvgSamplesPerSec=5.876385114794843, CurrSamplesPerSec=5.348985484882464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3204/5000 [11:06:38<5:37:52, 11.29s/it][2022-12-20 21:29:13,925] [INFO] [timer.py:197:stop] 0/6488, RunningAvgSamplesPerSec=5.87639712521979, CurrSamplesPerSec=5.357480911505666, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3205/5000 [11:06:49<5:34:35, 11.18s/it][2022-12-20 21:29:24,933] [INFO] [timer.py:197:stop] 0/6490, RunningAvgSamplesPerSec=5.876397998877807, CurrSamplesPerSec=5.342515833372143, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3206/5000 [11:07:00<5:32:49, 11.13s/it][2022-12-20 21:29:35,903] [INFO] [timer.py:197:stop] 0/6492, RunningAvgSamplesPerSec=5.8764053479878, CurrSamplesPerSec=5.3902156492231486, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3207/5000 [11:07:11<5:31:11, 11.08s/it][2022-12-20 21:29:46,902] [INFO] [timer.py:197:stop] 0/6494, RunningAvgSamplesPerSec=5.876407775323178, CurrSamplesPerSec=5.3565191821667595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3208/5000 [11:07:22<5:30:15, 11.06s/it][2022-12-20 21:29:57,913] [INFO] [timer.py:197:stop] 0/6496, RunningAvgSamplesPerSec=5.87640834462596, CurrSamplesPerSec=5.342688091917127, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3209/5000 [11:07:33<5:29:39, 11.04s/it][2022-12-20 21:30:08,898] [INFO] [timer.py:197:stop] 0/6498, RunningAvgSamplesPerSec=5.876413422244448, CurrSamplesPerSec=5.359413538781795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3210/5000 [11:07:44<5:28:56, 11.03s/it][2022-12-20 21:30:19,927] [INFO] [logging.py:68:log_dist] [Rank 0] step=3250, skipped=5, lr=[3.9022222222222225e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:30:19,929] [INFO] [timer.py:197:stop] 0/6500, RunningAvgSamplesPerSec=5.876410622343033, CurrSamplesPerSec=5.284151909773724, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3211/5000 [11:07:55<5:28:47, 11.03s/it][2022-12-20 21:30:30,912] [INFO] [timer.py:197:stop] 0/6502, RunningAvgSamplesPerSec=5.8764157984207674, CurrSamplesPerSec=5.345617548961401, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3212/5000 [11:08:06<5:28:13, 11.01s/it][2022-12-20 21:30:41,935] [INFO] [timer.py:197:stop] 0/6504, RunningAvgSamplesPerSec=5.8764145410436806, CurrSamplesPerSec=5.313141102465751, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3213/5000 [11:08:17<5:28:06, 11.02s/it][2022-12-20 21:30:52,942] [INFO] [timer.py:197:stop] 0/6506, RunningAvgSamplesPerSec=5.8764156496928885, CurrSamplesPerSec=5.338994837452263, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3214/5000 [11:08:28<5:27:50, 11.01s/it][2022-12-20 21:31:04,002] [INFO] [timer.py:197:stop] 0/6508, RunningAvgSamplesPerSec=5.87640809004743, CurrSamplesPerSec=5.303552223092157, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3215/5000 [11:08:39<5:28:04, 11.03s/it][2022-12-20 21:31:15,065] [INFO] [timer.py:197:stop] 0/6510, RunningAvgSamplesPerSec=5.876400242994232, CurrSamplesPerSec=5.305167641708319, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3216/5000 [11:08:50<5:28:12, 11.04s/it][2022-12-20 21:31:26,075] [INFO] [timer.py:197:stop] 0/6512, RunningAvgSamplesPerSec=5.876401022357505, CurrSamplesPerSec=5.3060151550928785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3217/5000 [11:09:01<5:27:46, 11.03s/it][2022-12-20 21:31:37,101] [INFO] [timer.py:197:stop] 0/6514, RunningAvgSamplesPerSec=5.876399312951286, CurrSamplesPerSec=5.31036351877559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3218/5000 [11:09:12<5:27:32, 11.03s/it][2022-12-20 21:31:48,155] [INFO] [timer.py:197:stop] 0/6516, RunningAvgSamplesPerSec=5.876392715259438, CurrSamplesPerSec=5.321550391875264, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3219/5000 [11:09:23<5:27:35, 11.04s/it][2022-12-20 21:31:59,210] [INFO] [timer.py:197:stop] 0/6518, RunningAvgSamplesPerSec=5.8763860515865485, CurrSamplesPerSec=5.301317290237658, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3220/5000 [11:09:34<5:27:34, 11.04s/it][2022-12-20 21:32:10,200] [INFO] [logging.py:68:log_dist] [Rank 0] step=3260, skipped=5, lr=[3.88e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:32:10,202] [INFO] [timer.py:197:stop] 0/6520, RunningAvgSamplesPerSec=5.876389748218032, CurrSamplesPerSec=5.362526984391314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3221/5000 [11:09:45<5:26:56, 11.03s/it][2022-12-20 21:32:21,180] [INFO] [timer.py:197:stop] 0/6522, RunningAvgSamplesPerSec=5.876395695277371, CurrSamplesPerSec=5.3575341609457325, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3222/5000 [11:09:56<5:26:19, 11.01s/it][2022-12-20 21:32:32,241] [INFO] [timer.py:197:stop] 0/6524, RunningAvgSamplesPerSec=5.876387794423687, CurrSamplesPerSec=5.304716625833253, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3223/5000 [11:10:07<5:26:34, 11.03s/it][2022-12-20 21:32:43,228] [INFO] [timer.py:197:stop] 0/6526, RunningAvgSamplesPerSec=5.876392398223364, CurrSamplesPerSec=5.3137812013751295, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3224/5000 [11:10:18<5:26:02, 11.01s/it][2022-12-20 21:32:54,302] [INFO] [timer.py:197:stop] 0/6528, RunningAvgSamplesPerSec=5.876382349249041, CurrSamplesPerSec=5.279760986657776, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 64%|██████▍   | 3225/5000 [11:10:29<5:26:22, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.8711111111111115e-06, 'epoch': 78.65}

 64%|██████▍   | 3225/5000 [11:10:29<5:26:22, 11.03s/it][2022-12-20 21:33:05,363] [INFO] [timer.py:197:stop] 0/6530, RunningAvgSamplesPerSec=5.876374938086388, CurrSamplesPerSec=5.3116614510824585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3226/5000 [11:10:40<5:26:26, 11.04s/it][2022-12-20 21:33:16,300] [INFO] [timer.py:197:stop] 0/6532, RunningAvgSamplesPerSec=5.876387565440981, CurrSamplesPerSec=5.339519037326742, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3227/5000 [11:10:51<5:25:20, 11.01s/it][2022-12-20 21:33:27,317] [INFO] [timer.py:197:stop] 0/6534, RunningAvgSamplesPerSec=5.876388430269386, CurrSamplesPerSec=5.311941463962921, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3228/5000 [11:11:02<5:25:13, 11.01s/it][2022-12-20 21:33:38,352] [INFO] [timer.py:197:stop] 0/6536, RunningAvgSamplesPerSec=5.876385016128062, CurrSamplesPerSec=5.301398325737773, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3229/5000 [11:11:13<5:25:14, 11.02s/it][2022-12-20 21:33:49,344] [INFO] [timer.py:197:stop] 0/6538, RunningAvgSamplesPerSec=5.876388471749313, CurrSamplesPerSec=5.308051677933515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3230/5000 [11:11:24<5:24:49, 11.01s/it][2022-12-20 21:34:00,303] [INFO] [logging.py:68:log_dist] [Rank 0] step=3270, skipped=5, lr=[3.857777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:34:00,304] [INFO] [timer.py:197:stop] 0/6540, RunningAvgSamplesPerSec=5.876397246598861, CurrSamplesPerSec=5.354624316198671, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3231/5000 [11:11:35<5:24:11, 11.00s/it][2022-12-20 21:34:11,372] [INFO] [timer.py:197:stop] 0/6542, RunningAvgSamplesPerSec=5.876388462193976, CurrSamplesPerSec=5.29607451721135, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3232/5000 [11:11:46<5:24:38, 11.02s/it][2022-12-20 21:34:22,392] [INFO] [timer.py:197:stop] 0/6544, RunningAvgSamplesPerSec=5.876387569397221, CurrSamplesPerSec=5.306126750894322, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3233/5000 [11:11:57<5:24:28, 11.02s/it][2022-12-20 21:34:33,368] [INFO] [timer.py:197:stop] 0/6546, RunningAvgSamplesPerSec=5.8763938087732495, CurrSamplesPerSec=5.337822344784141, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3234/5000 [11:12:08<5:23:55, 11.01s/it][2022-12-20 21:34:44,346] [INFO] [timer.py:197:stop] 0/6548, RunningAvgSamplesPerSec=5.876399626194584, CurrSamplesPerSec=5.325906936315606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3235/5000 [11:12:19<5:23:30, 11.00s/it][2022-12-20 21:34:55,362] [INFO] [timer.py:197:stop] 0/6550, RunningAvgSamplesPerSec=5.876399184821528, CurrSamplesPerSec=5.333549084383961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3236/5000 [11:12:30<5:23:29, 11.00s/it][2022-12-20 21:35:06,370] [INFO] [timer.py:197:stop] 0/6552, RunningAvgSamplesPerSec=5.87640043527191, CurrSamplesPerSec=5.3092819029180855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3237/5000 [11:12:41<5:23:20, 11.00s/it][2022-12-20 21:35:17,383] [INFO] [timer.py:197:stop] 0/6554, RunningAvgSamplesPerSec=5.876400762828286, CurrSamplesPerSec=5.330019785581214, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3238/5000 [11:12:52<5:23:14, 11.01s/it][2022-12-20 21:35:28,385] [INFO] [timer.py:197:stop] 0/6556, RunningAvgSamplesPerSec=5.876402775800579, CurrSamplesPerSec=5.3355609869674865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3239/5000 [11:13:03<5:23:00, 11.01s/it][2022-12-20 21:35:38,488] [INFO] [timer.py:197:stop] 0/6558, RunningAvgSamplesPerSec=5.876550910219189, CurrSamplesPerSec=5.316438744890436, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3240/5000 [11:13:18<5:58:03, 12.21s/it][2022-12-20 21:35:49,478] [INFO] [logging.py:68:log_dist] [Rank 0] step=3280, skipped=5, lr=[3.835555555555555e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:35:49,480] [INFO] [timer.py:197:stop] 0/6560, RunningAvgSamplesPerSec=5.876554339001798, CurrSamplesPerSec=5.3136370973051426, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3241/5000 [11:13:29<5:47:05, 11.84s/it][2022-12-20 21:36:00,470] [INFO] [timer.py:197:stop] 0/6562, RunningAvgSamplesPerSec=5.876557983145742, CurrSamplesPerSec=5.3071789608508455, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3242/5000 [11:13:41<5:40:01, 11.60s/it][2022-12-20 21:36:11,541] [INFO] [timer.py:197:stop] 0/6564, RunningAvgSamplesPerSec=5.876548197565742, CurrSamplesPerSec=5.294785234676136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3243/5000 [11:13:52<5:35:06, 11.44s/it][2022-12-20 21:36:22,562] [INFO] [timer.py:197:stop] 0/6566, RunningAvgSamplesPerSec=5.876546748445909, CurrSamplesPerSec=5.336025960988943, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3244/5000 [11:14:03<5:31:11, 11.32s/it][2022-12-20 21:36:33,584] [INFO] [timer.py:197:stop] 0/6568, RunningAvgSamplesPerSec=5.876545372900037, CurrSamplesPerSec=5.333681129246278, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3245/5000 [11:14:14<5:27:56, 11.21s/it][2022-12-20 21:36:44,590] [INFO] [timer.py:197:stop] 0/6570, RunningAvgSamplesPerSec=5.876546435656051, CurrSamplesPerSec=5.3005847328727835, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3246/5000 [11:14:25<5:26:25, 11.17s/it][2022-12-20 21:36:55,601] [INFO] [timer.py:197:stop] 0/6572, RunningAvgSamplesPerSec=5.876547010076143, CurrSamplesPerSec=5.344196147343519, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3247/5000 [11:14:36<5:24:34, 11.11s/it][2022-12-20 21:37:06,571] [INFO] [timer.py:197:stop] 0/6574, RunningAvgSamplesPerSec=5.876554167933507, CurrSamplesPerSec=5.34926475658042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3248/5000 [11:14:47<5:22:56, 11.06s/it][2022-12-20 21:37:17,558] [INFO] [timer.py:197:stop] 0/6576, RunningAvgSamplesPerSec=5.876558557123496, CurrSamplesPerSec=5.3119759420645245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▍   | 3249/5000 [11:14:58<5:22:48, 11.06s/it][2022-12-20 21:37:28,590] [INFO] [timer.py:197:stop] 0/6578, RunningAvgSamplesPerSec=5.876555537843151, CurrSamplesPerSec=5.342273839514454, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3250/5000 [11:15:09<5:21:53, 11.04s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.8155555555555555e-06, 'epoch': 79.27}

 65%|██████▌   | 3250/5000 [11:15:09<5:21:53, 11.04s/it][2022-12-20 21:37:39,582] [INFO] [logging.py:68:log_dist] [Rank 0] step=3290, skipped=5, lr=[3.813333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:37:39,584] [INFO] [timer.py:197:stop] 0/6580, RunningAvgSamplesPerSec=5.876558780307483, CurrSamplesPerSec=5.326824300524972, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3251/5000 [11:15:20<5:21:31, 11.03s/it][2022-12-20 21:37:50,600] [INFO] [timer.py:197:stop] 0/6582, RunningAvgSamplesPerSec=5.87655816160739, CurrSamplesPerSec=5.325796831577375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3252/5000 [11:15:31<5:21:04, 11.02s/it][2022-12-20 21:38:01,608] [INFO] [timer.py:197:stop] 0/6584, RunningAvgSamplesPerSec=5.8765590232739795, CurrSamplesPerSec=5.318052963743396, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3253/5000 [11:15:42<5:21:03, 11.03s/it][2022-12-20 21:38:12,623] [INFO] [timer.py:197:stop] 0/6586, RunningAvgSamplesPerSec=5.876558580498142, CurrSamplesPerSec=5.340697800917922, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3254/5000 [11:15:53<5:20:29, 11.01s/it][2022-12-20 21:38:23,592] [INFO] [timer.py:197:stop] 0/6588, RunningAvgSamplesPerSec=5.876565901240078, CurrSamplesPerSec=5.354105903231055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3255/5000 [11:16:04<5:20:07, 11.01s/it][2022-12-20 21:38:34,673] [INFO] [timer.py:197:stop] 0/6590, RunningAvgSamplesPerSec=5.876554843997878, CurrSamplesPerSec=5.275271610143361, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3256/5000 [11:16:15<5:20:34, 11.03s/it][2022-12-20 21:38:45,727] [INFO] [timer.py:197:stop] 0/6592, RunningAvgSamplesPerSec=5.876548124294219, CurrSamplesPerSec=5.297321357916446, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3257/5000 [11:16:26<5:20:34, 11.04s/it][2022-12-20 21:38:56,760] [INFO] [timer.py:197:stop] 0/6594, RunningAvgSamplesPerSec=5.8765449056449555, CurrSamplesPerSec=5.311292560261655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3258/5000 [11:16:37<5:20:24, 11.04s/it][2022-12-20 21:39:07,797] [INFO] [timer.py:197:stop] 0/6596, RunningAvgSamplesPerSec=5.8765410933148035, CurrSamplesPerSec=5.312962121129353, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3259/5000 [11:16:48<5:19:59, 11.03s/it][2022-12-20 21:39:18,786] [INFO] [timer.py:197:stop] 0/6598, RunningAvgSamplesPerSec=5.876545062811377, CurrSamplesPerSec=5.330784850417162, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3260/5000 [11:16:59<5:19:29, 11.02s/it][2022-12-20 21:39:29,806] [INFO] [logging.py:68:log_dist] [Rank 0] step=3300, skipped=5, lr=[3.7911111111111114e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:39:29,808] [INFO] [timer.py:197:stop] 0/6600, RunningAvgSamplesPerSec=5.876543300860448, CurrSamplesPerSec=5.30208565663978, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3261/5000 [11:17:10<5:19:37, 11.03s/it][2022-12-20 21:39:40,833] [INFO] [timer.py:197:stop] 0/6602, RunningAvgSamplesPerSec=5.876541238006011, CurrSamplesPerSec=5.327376139484198, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3262/5000 [11:17:21<5:19:24, 11.03s/it][2022-12-20 21:39:51,890] [INFO] [timer.py:197:stop] 0/6604, RunningAvgSamplesPerSec=5.8765339166448, CurrSamplesPerSec=5.299084449419083, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3263/5000 [11:17:32<5:19:33, 11.04s/it][2022-12-20 21:40:02,921] [INFO] [timer.py:197:stop] 0/6606, RunningAvgSamplesPerSec=5.876530956457931, CurrSamplesPerSec=5.329343815321253, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3264/5000 [11:17:43<5:19:10, 11.03s/it][2022-12-20 21:40:13,888] [INFO] [timer.py:197:stop] 0/6608, RunningAvgSamplesPerSec=5.8765383976262555, CurrSamplesPerSec=5.373233597618539, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3265/5000 [11:17:54<5:18:06, 11.00s/it][2022-12-20 21:40:24,847] [INFO] [timer.py:197:stop] 0/6610, RunningAvgSamplesPerSec=5.876547153461613, CurrSamplesPerSec=5.346157317663753, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3266/5000 [11:18:05<5:17:58, 11.00s/it][2022-12-20 21:40:35,879] [INFO] [timer.py:197:stop] 0/6612, RunningAvgSamplesPerSec=5.876543951073196, CurrSamplesPerSec=5.322873219958154, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3267/5000 [11:18:16<5:18:03, 11.01s/it][2022-12-20 21:40:46,883] [INFO] [timer.py:197:stop] 0/6614, RunningAvgSamplesPerSec=5.876545375620576, CurrSamplesPerSec=5.349590112903691, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3268/5000 [11:18:27<5:17:42, 11.01s/it][2022-12-20 21:40:57,905] [INFO] [timer.py:197:stop] 0/6616, RunningAvgSamplesPerSec=5.876543873461056, CurrSamplesPerSec=5.324003040548273, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3269/5000 [11:18:38<5:17:49, 11.02s/it][2022-12-20 21:41:08,944] [INFO] [timer.py:197:stop] 0/6618, RunningAvgSamplesPerSec=5.876539577107041, CurrSamplesPerSec=5.325619954993742, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3270/5000 [11:18:49<5:17:39, 11.02s/it][2022-12-20 21:41:19,987] [INFO] [logging.py:68:log_dist] [Rank 0] step=3310, skipped=5, lr=[3.768888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:41:19,989] [INFO] [timer.py:197:stop] 0/6620, RunningAvgSamplesPerSec=5.876534285353074, CurrSamplesPerSec=5.301392253224249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3271/5000 [11:19:00<5:17:53, 11.03s/it][2022-12-20 21:41:31,013] [INFO] [timer.py:197:stop] 0/6622, RunningAvgSamplesPerSec=5.876532513786425, CurrSamplesPerSec=5.338259049788918, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3272/5000 [11:19:11<5:17:18, 11.02s/it][2022-12-20 21:41:42,015] [INFO] [timer.py:197:stop] 0/6624, RunningAvgSamplesPerSec=5.876534237102872, CurrSamplesPerSec=5.323622931503264, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3273/5000 [11:19:22<5:17:20, 11.03s/it][2022-12-20 21:41:52,976] [INFO] [timer.py:197:stop] 0/6626, RunningAvgSamplesPerSec=5.876542765766015, CurrSamplesPerSec=5.397361336496472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 65%|██████▌   | 3274/5000 [11:19:33<5:16:06, 10.99s/it][2022-12-20 21:42:03,900] [INFO] [timer.py:197:stop] 0/6628, RunningAvgSamplesPerSec=5.876557352650253, CurrSamplesPerSec=5.378238498100474, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3275/5000 [11:19:44<5:15:19, 10.97s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.7600000000000004e-06, 'epoch': 79.87}

 66%|██████▌   | 3275/5000 [11:19:44<5:15:19, 10.97s/it][2022-12-20 21:42:14,870] [INFO] [timer.py:197:stop] 0/6630, RunningAvgSamplesPerSec=5.876564822090592, CurrSamplesPerSec=5.335096306041123, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3276/5000 [11:19:55<5:15:27, 10.98s/it][2022-12-20 21:42:25,822] [INFO] [timer.py:197:stop] 0/6632, RunningAvgSamplesPerSec=5.876574575337659, CurrSamplesPerSec=5.379633002775436, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3277/5000 [11:20:06<5:15:13, 10.98s/it][2022-12-20 21:42:36,886] [INFO] [timer.py:197:stop] 0/6634, RunningAvgSamplesPerSec=5.8765665002906635, CurrSamplesPerSec=5.299294718343364, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3278/5000 [11:20:17<5:15:20, 10.99s/it][2022-12-20 21:42:47,839] [INFO] [timer.py:197:stop] 0/6636, RunningAvgSamplesPerSec=5.8765763268209, CurrSamplesPerSec=5.351635903011341, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3279/5000 [11:20:28<5:14:56, 10.98s/it][2022-12-20 21:42:58,839] [INFO] [timer.py:197:stop] 0/6638, RunningAvgSamplesPerSec=5.876578632059887, CurrSamplesPerSec=5.317842046403018, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3280/5000 [11:20:39<5:15:08, 10.99s/it][2022-12-20 21:43:08,931] [INFO] [logging.py:68:log_dist] [Rank 0] step=3320, skipped=5, lr=[3.7466666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:43:08,932] [INFO] [timer.py:197:stop] 0/6640, RunningAvgSamplesPerSec=5.876726397002825, CurrSamplesPerSec=6.277170043707559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 21:43:19,929] [INFO] [timer.py:197:stop] 0/6642, RunningAvgSamplesPerSec=5.876728559817958, CurrSamplesPerSec=5.360211255484006, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3281/5000 [11:20:55<5:59:15, 12.54s/it][2022-12-20 21:43:30,934] [INFO] [timer.py:197:stop] 0/6644, RunningAvgSamplesPerSec=5.876730206125147, CurrSamplesPerSec=5.316164153506739, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3282/5000 [11:21:06<5:45:51, 12.08s/it][2022-12-20 21:43:41,942] [INFO] [timer.py:197:stop] 0/6646, RunningAvgSamplesPerSec=5.876731119625821, CurrSamplesPerSec=5.328501103235164, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3283/5000 [11:21:17<5:36:28, 11.76s/it][2022-12-20 21:43:52,920] [INFO] [timer.py:197:stop] 0/6648, RunningAvgSamplesPerSec=5.876736925053151, CurrSamplesPerSec=5.332237045189865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3284/5000 [11:21:28<5:29:34, 11.52s/it][2022-12-20 21:44:03,896] [INFO] [timer.py:197:stop] 0/6650, RunningAvgSamplesPerSec=5.876742971772372, CurrSamplesPerSec=5.34092966299962, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3285/5000 [11:21:39<5:24:41, 11.36s/it][2022-12-20 21:44:14,840] [INFO] [timer.py:197:stop] 0/6652, RunningAvgSamplesPerSec=5.87675413179437, CurrSamplesPerSec=5.4003378185356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3286/5000 [11:21:50<5:20:56, 11.23s/it][2022-12-20 21:44:25,842] [INFO] [timer.py:197:stop] 0/6654, RunningAvgSamplesPerSec=5.876756032385161, CurrSamplesPerSec=5.303465254029104, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3287/5000 [11:22:01<5:18:45, 11.16s/it][2022-12-20 21:44:36,826] [INFO] [timer.py:197:stop] 0/6656, RunningAvgSamplesPerSec=5.876760816521527, CurrSamplesPerSec=5.368887152297971, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3288/5000 [11:22:12<5:17:01, 11.11s/it][2022-12-20 21:44:47,737] [INFO] [timer.py:197:stop] 0/6658, RunningAvgSamplesPerSec=5.876777452096817, CurrSamplesPerSec=5.363080459795835, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3289/5000 [11:22:23<5:15:08, 11.05s/it][2022-12-20 21:44:58,818] [INFO] [logging.py:68:log_dist] [Rank 0] step=3330, skipped=5, lr=[3.724444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:44:58,819] [INFO] [timer.py:197:stop] 0/6660, RunningAvgSamplesPerSec=5.876766480894791, CurrSamplesPerSec=5.266381226222015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3290/5000 [11:22:34<5:15:13, 11.06s/it][2022-12-20 21:45:09,869] [INFO] [timer.py:197:stop] 0/6662, RunningAvgSamplesPerSec=5.876760939341473, CurrSamplesPerSec=5.31578642613019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3291/5000 [11:22:45<5:14:56, 11.06s/it][2022-12-20 21:45:20,839] [INFO] [timer.py:197:stop] 0/6664, RunningAvgSamplesPerSec=5.876768163474889, CurrSamplesPerSec=5.329391851379955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3292/5000 [11:22:56<5:14:00, 11.03s/it][2022-12-20 21:45:31,887] [INFO] [timer.py:197:stop] 0/6666, RunningAvgSamplesPerSec=5.876762712738285, CurrSamplesPerSec=5.302318995944538, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3293/5000 [11:23:07<5:13:58, 11.04s/it][2022-12-20 21:45:42,920] [INFO] [timer.py:197:stop] 0/6668, RunningAvgSamplesPerSec=5.876759638250988, CurrSamplesPerSec=5.296096877834512, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3294/5000 [11:23:18<5:13:46, 11.04s/it][2022-12-20 21:45:53,927] [INFO] [timer.py:197:stop] 0/6670, RunningAvgSamplesPerSec=5.876760616592619, CurrSamplesPerSec=5.3396084674368085, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3295/5000 [11:23:29<5:13:20, 11.03s/it][2022-12-20 21:46:04,959] [INFO] [timer.py:197:stop] 0/6672, RunningAvgSamplesPerSec=5.876757649743471, CurrSamplesPerSec=5.3000457552300135, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3296/5000 [11:23:40<5:13:12, 11.03s/it][2022-12-20 21:46:16,016] [INFO] [timer.py:197:stop] 0/6674, RunningAvgSamplesPerSec=5.876750535936039, CurrSamplesPerSec=5.272061271014854, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3297/5000 [11:23:51<5:13:16, 11.04s/it][2022-12-20 21:46:27,054] [INFO] [timer.py:197:stop] 0/6676, RunningAvgSamplesPerSec=5.876746739841108, CurrSamplesPerSec=5.320910318427502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3298/5000 [11:24:02<5:13:05, 11.04s/it][2022-12-20 21:46:38,078] [INFO] [timer.py:197:stop] 0/6678, RunningAvgSamplesPerSec=5.876745132964561, CurrSamplesPerSec=5.329512474375459, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3299/5000 [11:24:13<5:12:47, 11.03s/it][2022-12-20 21:46:49,105] [INFO] [logging.py:68:log_dist] [Rank 0] step=3340, skipped=5, lr=[3.7022222222222227e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:46:49,107] [INFO] [timer.py:197:stop] 0/6680, RunningAvgSamplesPerSec=5.876742562181444, CurrSamplesPerSec=5.3035467743442135, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3300/5000 [11:24:24<5:12:34, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.7022222222222227e-06, 'epoch': 80.48}

 66%|██████▌   | 3300/5000 [11:24:24<5:12:34, 11.03s/it][2022-12-20 21:47:00,032] [INFO] [timer.py:197:stop] 0/6682, RunningAvgSamplesPerSec=5.876757031091633, CurrSamplesPerSec=5.352762384414847, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3301/5000 [11:24:35<5:11:29, 11.00s/it][2022-12-20 21:47:11,041] [INFO] [timer.py:197:stop] 0/6684, RunningAvgSamplesPerSec=5.87675779998473, CurrSamplesPerSec=5.332021823919215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3302/5000 [11:24:46<5:11:22, 11.00s/it][2022-12-20 21:47:22,067] [INFO] [timer.py:197:stop] 0/6686, RunningAvgSamplesPerSec=5.876755966507399, CurrSamplesPerSec=5.325087916817084, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3303/5000 [11:24:57<5:11:23, 11.01s/it][2022-12-20 21:47:33,079] [INFO] [timer.py:197:stop] 0/6688, RunningAvgSamplesPerSec=5.876756270043796, CurrSamplesPerSec=5.331858724883191, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3304/5000 [11:25:08<5:11:13, 11.01s/it][2022-12-20 21:47:44,075] [INFO] [timer.py:197:stop] 0/6690, RunningAvgSamplesPerSec=5.876759072874179, CurrSamplesPerSec=5.326167951339064, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3305/5000 [11:25:19<5:10:55, 11.01s/it][2022-12-20 21:47:55,125] [INFO] [timer.py:197:stop] 0/6692, RunningAvgSamplesPerSec=5.8767533765865725, CurrSamplesPerSec=5.31607719144456, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3306/5000 [11:25:30<5:11:06, 11.02s/it][2022-12-20 21:48:06,160] [INFO] [timer.py:197:stop] 0/6694, RunningAvgSamplesPerSec=5.876750000661042, CurrSamplesPerSec=5.332687032403595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3307/5000 [11:25:41<5:11:03, 11.02s/it][2022-12-20 21:48:17,161] [INFO] [timer.py:197:stop] 0/6696, RunningAvgSamplesPerSec=5.876752343569615, CurrSamplesPerSec=5.357814326036864, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3308/5000 [11:25:52<5:10:40, 11.02s/it][2022-12-20 21:48:28,203] [INFO] [timer.py:197:stop] 0/6698, RunningAvgSamplesPerSec=5.876747705340313, CurrSamplesPerSec=5.308215842939329, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3309/5000 [11:26:03<5:10:42, 11.02s/it][2022-12-20 21:48:39,180] [INFO] [logging.py:68:log_dist] [Rank 0] step=3350, skipped=5, lr=[3.6800000000000003e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:48:39,182] [INFO] [timer.py:197:stop] 0/6700, RunningAvgSamplesPerSec=5.876753085715051, CurrSamplesPerSec=5.303998429718006, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3310/5000 [11:26:14<5:10:08, 11.01s/it][2022-12-20 21:48:50,205] [INFO] [timer.py:197:stop] 0/6702, RunningAvgSamplesPerSec=5.876751489110395, CurrSamplesPerSec=5.308860425236555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3311/5000 [11:26:25<5:10:03, 11.01s/it][2022-12-20 21:49:01,244] [INFO] [timer.py:197:stop] 0/6704, RunningAvgSamplesPerSec=5.876747369995446, CurrSamplesPerSec=5.308913132536044, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▌   | 3312/5000 [11:26:36<5:10:05, 11.02s/it][2022-12-20 21:49:12,254] [INFO] [timer.py:197:stop] 0/6706, RunningAvgSamplesPerSec=5.876748017035954, CurrSamplesPerSec=5.3470558963244805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3313/5000 [11:26:47<5:09:47, 11.02s/it][2022-12-20 21:49:23,296] [INFO] [timer.py:197:stop] 0/6708, RunningAvgSamplesPerSec=5.876743598750022, CurrSamplesPerSec=5.323761876182243, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3314/5000 [11:26:58<5:09:48, 11.03s/it][2022-12-20 21:49:34,284] [INFO] [timer.py:197:stop] 0/6710, RunningAvgSamplesPerSec=5.876747493184284, CurrSamplesPerSec=5.337908533695213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3315/5000 [11:27:09<5:09:18, 11.01s/it][2022-12-20 21:49:45,270] [INFO] [timer.py:197:stop] 0/6712, RunningAvgSamplesPerSec=5.87675190671935, CurrSamplesPerSec=5.333694270505217, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3316/5000 [11:27:20<5:08:53, 11.01s/it][2022-12-20 21:49:56,309] [INFO] [timer.py:197:stop] 0/6714, RunningAvgSamplesPerSec=5.876747995074545, CurrSamplesPerSec=5.313293593140977, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3317/5000 [11:27:31<5:08:59, 11.02s/it][2022-12-20 21:50:07,292] [INFO] [timer.py:197:stop] 0/6716, RunningAvgSamplesPerSec=5.87675279207203, CurrSamplesPerSec=5.330138320264769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3318/5000 [11:27:42<5:08:31, 11.01s/it][2022-12-20 21:50:18,248] [INFO] [timer.py:197:stop] 0/6718, RunningAvgSamplesPerSec=5.876761904722673, CurrSamplesPerSec=5.361325288359859, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3319/5000 [11:27:53<5:07:55, 10.99s/it][2022-12-20 21:50:29,213] [INFO] [logging.py:68:log_dist] [Rank 0] step=3360, skipped=5, lr=[3.657777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:50:29,215] [INFO] [timer.py:197:stop] 0/6720, RunningAvgSamplesPerSec=5.876769155030542, CurrSamplesPerSec=5.3359264684004994, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3320/5000 [11:28:04<5:07:32, 10.98s/it][2022-12-20 21:50:40,239] [INFO] [timer.py:197:stop] 0/6722, RunningAvgSamplesPerSec=5.8767675753111, CurrSamplesPerSec=5.338921780488556, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3321/5000 [11:28:15<5:07:42, 11.00s/it][2022-12-20 21:50:50,322] [INFO] [timer.py:197:stop] 0/6724, RunningAvgSamplesPerSec=5.876915327060252, CurrSamplesPerSec=5.354080914289216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3322/5000 [11:28:30<5:41:08, 12.20s/it][2022-12-20 21:51:01,272] [INFO] [timer.py:197:stop] 0/6726, RunningAvgSamplesPerSec=5.87692531356035, CurrSamplesPerSec=5.3646590170285045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3323/5000 [11:28:41<5:30:50, 11.84s/it][2022-12-20 21:51:12,322] [INFO] [timer.py:197:stop] 0/6728, RunningAvgSamplesPerSec=5.876919204131609, CurrSamplesPerSec=5.313316309742369, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3324/5000 [11:28:52<5:23:49, 11.59s/it][2022-12-20 21:51:23,332] [INFO] [timer.py:197:stop] 0/6730, RunningAvgSamplesPerSec=5.876919415076658, CurrSamplesPerSec=5.32425985604738, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 66%|██████▋   | 3325/5000 [11:29:03<5:18:52, 11.42s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.6466666666666668e-06, 'epoch': 81.1}

 66%|██████▋   | 3325/5000 [11:29:03<5:18:52, 11.42s/it][2022-12-20 21:51:34,370] [INFO] [timer.py:197:stop] 0/6732, RunningAvgSamplesPerSec=5.876915537877461, CurrSamplesPerSec=5.314782151422378, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3326/5000 [11:29:14<5:15:32, 11.31s/it][2022-12-20 21:51:45,411] [INFO] [timer.py:197:stop] 0/6734, RunningAvgSamplesPerSec=5.876910843716765, CurrSamplesPerSec=5.31816211634895, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3327/5000 [11:29:25<5:13:00, 11.23s/it][2022-12-20 21:51:56,433] [INFO] [timer.py:197:stop] 0/6736, RunningAvgSamplesPerSec=5.876909317866746, CurrSamplesPerSec=5.326181689680577, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3328/5000 [11:29:36<5:10:39, 11.15s/it][2022-12-20 21:52:07,442] [INFO] [timer.py:197:stop] 0/6738, RunningAvgSamplesPerSec=5.876909771595456, CurrSamplesPerSec=5.288438034450293, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3329/5000 [11:29:47<5:09:39, 11.12s/it][2022-12-20 21:52:18,460] [INFO] [logging.py:68:log_dist] [Rank 0] step=3370, skipped=5, lr=[3.635555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:52:18,461] [INFO] [timer.py:197:stop] 0/6740, RunningAvgSamplesPerSec=5.876908507086498, CurrSamplesPerSec=5.314929895553247, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3330/5000 [11:29:58<5:08:28, 11.08s/it][2022-12-20 21:52:29,452] [INFO] [timer.py:197:stop] 0/6742, RunningAvgSamplesPerSec=5.876911926709899, CurrSamplesPerSec=5.323124015183574, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3331/5000 [11:30:09<5:07:41, 11.06s/it][2022-12-20 21:52:40,433] [INFO] [timer.py:197:stop] 0/6744, RunningAvgSamplesPerSec=5.876916954619886, CurrSamplesPerSec=5.35051010893587, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3332/5000 [11:30:20<5:07:03, 11.05s/it][2022-12-20 21:52:51,422] [INFO] [timer.py:197:stop] 0/6746, RunningAvgSamplesPerSec=5.876920663450869, CurrSamplesPerSec=5.366228849025981, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3333/5000 [11:30:31<5:06:01, 11.01s/it][2022-12-20 21:53:02,422] [INFO] [timer.py:197:stop] 0/6748, RunningAvgSamplesPerSec=5.876922843960271, CurrSamplesPerSec=5.316866060441772, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3334/5000 [11:30:42<5:05:32, 11.00s/it][2022-12-20 21:53:13,345] [INFO] [timer.py:197:stop] 0/6750, RunningAvgSamplesPerSec=5.876937271852891, CurrSamplesPerSec=5.36610376929793, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3335/5000 [11:30:53<5:04:46, 10.98s/it][2022-12-20 21:53:24,296] [INFO] [timer.py:197:stop] 0/6752, RunningAvgSamplesPerSec=5.876947297139029, CurrSamplesPerSec=5.351655321087609, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3336/5000 [11:31:04<5:04:49, 10.99s/it][2022-12-20 21:53:35,329] [INFO] [timer.py:197:stop] 0/6754, RunningAvgSamplesPerSec=5.876943987266448, CurrSamplesPerSec=5.33209850509613, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3337/5000 [11:31:15<5:04:45, 11.00s/it][2022-12-20 21:53:46,284] [INFO] [timer.py:197:stop] 0/6756, RunningAvgSamplesPerSec=5.876953159231845, CurrSamplesPerSec=5.377035782531397, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3338/5000 [11:31:26<5:03:53, 10.97s/it][2022-12-20 21:53:57,248] [INFO] [timer.py:197:stop] 0/6758, RunningAvgSamplesPerSec=5.876960977729, CurrSamplesPerSec=5.3324774950106, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3339/5000 [11:31:37<5:04:10, 10.99s/it][2022-12-20 21:54:08,247] [INFO] [logging.py:68:log_dist] [Rank 0] step=3380, skipped=5, lr=[3.6133333333333336e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:54:08,248] [INFO] [timer.py:197:stop] 0/6760, RunningAvgSamplesPerSec=5.876963161466556, CurrSamplesPerSec=5.355239407632347, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3340/5000 [11:31:48<5:03:33, 10.97s/it][2022-12-20 21:54:19,201] [INFO] [timer.py:197:stop] 0/6762, RunningAvgSamplesPerSec=5.8769726984845985, CurrSamplesPerSec=5.340655298595116, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3341/5000 [11:31:59<5:03:25, 10.97s/it][2022-12-20 21:54:30,166] [INFO] [timer.py:197:stop] 0/6764, RunningAvgSamplesPerSec=5.876980176540531, CurrSamplesPerSec=5.350759462307742, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3342/5000 [11:32:10<5:03:29, 10.98s/it][2022-12-20 21:54:41,182] [INFO] [timer.py:197:stop] 0/6766, RunningAvgSamplesPerSec=5.876979711191236, CurrSamplesPerSec=5.340551595766481, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3343/5000 [11:32:21<5:03:05, 10.98s/it][2022-12-20 21:54:52,148] [INFO] [timer.py:197:stop] 0/6768, RunningAvgSamplesPerSec=5.876987266749327, CurrSamplesPerSec=5.333559681643645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3344/5000 [11:32:32<5:02:36, 10.96s/it][2022-12-20 21:55:03,075] [INFO] [timer.py:197:stop] 0/6770, RunningAvgSamplesPerSec=5.8770011927924815, CurrSamplesPerSec=5.344420226508184, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3345/5000 [11:32:43<5:02:44, 10.98s/it][2022-12-20 21:55:14,059] [INFO] [timer.py:197:stop] 0/6772, RunningAvgSamplesPerSec=5.8770058066882, CurrSamplesPerSec=5.359845008325232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3346/5000 [11:32:54<5:02:46, 10.98s/it][2022-12-20 21:55:25,058] [INFO] [timer.py:197:stop] 0/6774, RunningAvgSamplesPerSec=5.877008053417263, CurrSamplesPerSec=5.36154866452755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3347/5000 [11:33:05<5:02:40, 10.99s/it][2022-12-20 21:55:36,105] [INFO] [timer.py:197:stop] 0/6776, RunningAvgSamplesPerSec=5.877002772526877, CurrSamplesPerSec=5.314598640236406, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3348/5000 [11:33:16<5:02:34, 10.99s/it][2022-12-20 21:55:47,100] [INFO] [timer.py:197:stop] 0/6778, RunningAvgSamplesPerSec=5.877005866219796, CurrSamplesPerSec=5.315986652853264, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3349/5000 [11:33:27<5:02:41, 11.00s/it][2022-12-20 21:55:58,083] [INFO] [logging.py:68:log_dist] [Rank 0] step=3390, skipped=5, lr=[3.5911111111111112e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:55:58,085] [INFO] [timer.py:197:stop] 0/6780, RunningAvgSamplesPerSec=5.877010222566821, CurrSamplesPerSec=5.351602401826418, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3350/5000 [11:33:38<5:02:15, 10.99s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.5911111111111112e-06, 'epoch': 81.7}

 67%|██████▋   | 3350/5000 [11:33:38<5:02:15, 10.99s/it][2022-12-20 21:56:09,093] [INFO] [timer.py:197:stop] 0/6782, RunningAvgSamplesPerSec=5.877011138842423, CurrSamplesPerSec=5.318804058036418, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3351/5000 [11:33:49<5:02:39, 11.01s/it][2022-12-20 21:56:20,139] [INFO] [timer.py:197:stop] 0/6784, RunningAvgSamplesPerSec=5.877006093928275, CurrSamplesPerSec=5.3323476282292, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3352/5000 [11:34:00<5:02:09, 11.00s/it][2022-12-20 21:56:31,091] [INFO] [timer.py:197:stop] 0/6786, RunningAvgSamplesPerSec=5.8770159787752165, CurrSamplesPerSec=5.352823865782948, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3353/5000 [11:34:11<5:01:46, 10.99s/it][2022-12-20 21:56:42,082] [INFO] [timer.py:197:stop] 0/6788, RunningAvgSamplesPerSec=5.8770198090831665, CurrSamplesPerSec=5.340828712322784, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3354/5000 [11:34:22<5:01:31, 10.99s/it][2022-12-20 21:56:53,088] [INFO] [timer.py:197:stop] 0/6790, RunningAvgSamplesPerSec=5.877021013294528, CurrSamplesPerSec=5.320727649374366, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3355/5000 [11:34:33<5:01:22, 10.99s/it][2022-12-20 21:57:04,105] [INFO] [timer.py:197:stop] 0/6792, RunningAvgSamplesPerSec=5.877020632971732, CurrSamplesPerSec=5.302481340160647, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3356/5000 [11:34:44<5:01:47, 11.01s/it][2022-12-20 21:57:15,096] [INFO] [timer.py:197:stop] 0/6794, RunningAvgSamplesPerSec=5.877024227980055, CurrSamplesPerSec=5.367933346272797, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3357/5000 [11:34:55<5:01:19, 11.00s/it][2022-12-20 21:57:26,059] [INFO] [timer.py:197:stop] 0/6796, RunningAvgSamplesPerSec=5.877032077616335, CurrSamplesPerSec=5.382901030909974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3358/5000 [11:35:06<5:00:57, 11.00s/it][2022-12-20 21:57:37,103] [INFO] [timer.py:197:stop] 0/6798, RunningAvgSamplesPerSec=5.877026958059406, CurrSamplesPerSec=5.326881170596992, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3359/5000 [11:35:17<5:01:00, 11.01s/it][2022-12-20 21:57:48,142] [INFO] [logging.py:68:log_dist] [Rank 0] step=3400, skipped=5, lr=[3.568888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:57:48,143] [INFO] [timer.py:197:stop] 0/6800, RunningAvgSamplesPerSec=5.877022429298849, CurrSamplesPerSec=5.314393888699931, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3360/5000 [11:35:28<5:01:05, 11.02s/it][2022-12-20 21:57:59,169] [INFO] [timer.py:197:stop] 0/6802, RunningAvgSamplesPerSec=5.877020192535571, CurrSamplesPerSec=5.325660950431907, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3361/5000 [11:35:39<5:00:58, 11.02s/it][2022-12-20 21:58:10,218] [INFO] [timer.py:197:stop] 0/6804, RunningAvgSamplesPerSec=5.877014102959561, CurrSamplesPerSec=5.302982470426443, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3362/5000 [11:35:50<5:01:03, 11.03s/it][2022-12-20 21:58:20,341] [INFO] [timer.py:197:stop] 0/6806, RunningAvgSamplesPerSec=5.87715343379233, CurrSamplesPerSec=6.253252729519738, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 21:58:31,343] [INFO] [timer.py:197:stop] 0/6808, RunningAvgSamplesPerSec=5.877154685523359, CurrSamplesPerSec=5.323364066081431, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3363/5000 [11:36:06<5:42:58, 12.57s/it][2022-12-20 21:58:42,329] [INFO] [timer.py:197:stop] 0/6810, RunningAvgSamplesPerSec=5.877158908485091, CurrSamplesPerSec=5.327855972857088, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3364/5000 [11:36:17<5:29:48, 12.10s/it][2022-12-20 21:58:53,367] [INFO] [timer.py:197:stop] 0/6812, RunningAvgSamplesPerSec=5.877155035318235, CurrSamplesPerSec=5.324107791246855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3365/5000 [11:36:28<5:20:57, 11.78s/it][2022-12-20 21:59:04,351] [INFO] [timer.py:197:stop] 0/6814, RunningAvgSamplesPerSec=5.877159614862972, CurrSamplesPerSec=5.323887312714259, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3366/5000 [11:36:39<5:14:16, 11.54s/it][2022-12-20 21:59:15,357] [INFO] [timer.py:197:stop] 0/6816, RunningAvgSamplesPerSec=5.8771608161028945, CurrSamplesPerSec=5.321547859969634, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3367/5000 [11:36:50<5:09:43, 11.38s/it][2022-12-20 21:59:26,255] [INFO] [timer.py:197:stop] 0/6818, RunningAvgSamplesPerSec=5.877178988848708, CurrSamplesPerSec=5.39475109406134, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3368/5000 [11:37:01<5:05:35, 11.24s/it][2022-12-20 21:59:37,272] [INFO] [logging.py:68:log_dist] [Rank 0] step=3410, skipped=5, lr=[3.5466666666666673e-06], mom=[[0.9, 0.999]]
[2022-12-20 21:59:37,274] [INFO] [timer.py:197:stop] 0/6820, RunningAvgSamplesPerSec=5.877178042302333, CurrSamplesPerSec=5.303204363952804, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3369/5000 [11:37:12<5:03:38, 11.17s/it][2022-12-20 21:59:48,311] [INFO] [timer.py:197:stop] 0/6822, RunningAvgSamplesPerSec=5.877174079218388, CurrSamplesPerSec=5.313250894869602, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3370/5000 [11:37:23<5:02:22, 11.13s/it][2022-12-20 21:59:59,362] [INFO] [timer.py:197:stop] 0/6824, RunningAvgSamplesPerSec=5.8771680926985015, CurrSamplesPerSec=5.316782445241601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3371/5000 [11:37:34<5:01:32, 11.11s/it][2022-12-20 22:00:10,305] [INFO] [timer.py:197:stop] 0/6826, RunningAvgSamplesPerSec=5.877179085885858, CurrSamplesPerSec=5.357220667423125, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3372/5000 [11:37:45<5:00:01, 11.06s/it][2022-12-20 22:00:21,348] [INFO] [timer.py:197:stop] 0/6828, RunningAvgSamplesPerSec=5.877174308869121, CurrSamplesPerSec=5.312875684402559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3373/5000 [11:37:56<4:59:43, 11.05s/it][2022-12-20 22:00:32,359] [INFO] [timer.py:197:stop] 0/6830, RunningAvgSamplesPerSec=5.87717456571719, CurrSamplesPerSec=5.329425286680841, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 67%|██████▋   | 3374/5000 [11:38:07<4:59:12, 11.04s/it][2022-12-20 22:00:43,334] [INFO] [timer.py:197:stop] 0/6832, RunningAvgSamplesPerSec=5.877180575723231, CurrSamplesPerSec=5.344966139014561, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3375/5000 [11:38:18<4:58:28, 11.02s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.5333333333333335e-06, 'epoch': 82.31}

 68%|██████▊   | 3375/5000 [11:38:18<4:58:28, 11.02s/it][2022-12-20 22:00:54,342] [INFO] [timer.py:197:stop] 0/6834, RunningAvgSamplesPerSec=5.877181478525889, CurrSamplesPerSec=5.330518725203653, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3376/5000 [11:38:29<4:58:11, 11.02s/it][2022-12-20 22:01:05,332] [INFO] [timer.py:197:stop] 0/6836, RunningAvgSamplesPerSec=5.877185312715307, CurrSamplesPerSec=5.347984823132789, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3377/5000 [11:38:40<4:57:47, 11.01s/it][2022-12-20 22:01:16,320] [INFO] [timer.py:197:stop] 0/6838, RunningAvgSamplesPerSec=5.877189140223879, CurrSamplesPerSec=5.312320957959642, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3378/5000 [11:38:51<4:57:26, 11.00s/it][2022-12-20 22:01:27,298] [INFO] [logging.py:68:log_dist] [Rank 0] step=3420, skipped=5, lr=[3.524444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:01:27,299] [INFO] [timer.py:197:stop] 0/6840, RunningAvgSamplesPerSec=5.877194666974939, CurrSamplesPerSec=5.344442784426168, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3379/5000 [11:39:02<4:57:03, 11.00s/it][2022-12-20 22:01:38,343] [INFO] [timer.py:197:stop] 0/6842, RunningAvgSamplesPerSec=5.877189747891343, CurrSamplesPerSec=5.306849090616358, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3380/5000 [11:39:13<4:57:16, 11.01s/it][2022-12-20 22:01:49,297] [INFO] [timer.py:197:stop] 0/6844, RunningAvgSamplesPerSec=5.877199091220301, CurrSamplesPerSec=5.389259011321249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3381/5000 [11:39:24<4:56:38, 10.99s/it][2022-12-20 22:02:00,357] [INFO] [timer.py:197:stop] 0/6846, RunningAvgSamplesPerSec=5.87719174167605, CurrSamplesPerSec=5.309576998121164, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3382/5000 [11:39:35<4:56:59, 11.01s/it][2022-12-20 22:02:11,398] [INFO] [timer.py:197:stop] 0/6848, RunningAvgSamplesPerSec=5.877187428854836, CurrSamplesPerSec=5.302465838477172, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3383/5000 [11:39:46<4:57:01, 11.02s/it][2022-12-20 22:02:22,441] [INFO] [timer.py:197:stop] 0/6850, RunningAvgSamplesPerSec=5.877182739670573, CurrSamplesPerSec=5.336479346258684, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3384/5000 [11:39:58<4:57:01, 11.03s/it][2022-12-20 22:02:33,462] [INFO] [timer.py:197:stop] 0/6852, RunningAvgSamplesPerSec=5.877181529855066, CurrSamplesPerSec=5.320433422480445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3385/5000 [11:40:09<4:56:46, 11.03s/it][2022-12-20 22:02:44,444] [INFO] [timer.py:197:stop] 0/6854, RunningAvgSamplesPerSec=5.877186436927475, CurrSamplesPerSec=5.338868475651671, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3386/5000 [11:40:20<4:56:14, 11.01s/it][2022-12-20 22:02:55,471] [INFO] [timer.py:197:stop] 0/6856, RunningAvgSamplesPerSec=5.877184222371062, CurrSamplesPerSec=5.334600325669707, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3387/5000 [11:40:31<4:56:10, 11.02s/it][2022-12-20 22:03:06,405] [INFO] [timer.py:197:stop] 0/6858, RunningAvgSamplesPerSec=5.877196842011702, CurrSamplesPerSec=5.365163390818276, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3388/5000 [11:40:42<4:55:19, 10.99s/it][2022-12-20 22:03:17,400] [INFO] [logging.py:68:log_dist] [Rank 0] step=3430, skipped=5, lr=[3.5022222222222225e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:03:17,402] [INFO] [timer.py:197:stop] 0/6860, RunningAvgSamplesPerSec=5.87719921296691, CurrSamplesPerSec=5.34382655354221, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3389/5000 [11:40:52<4:55:10, 10.99s/it][2022-12-20 22:03:28,457] [INFO] [timer.py:197:stop] 0/6862, RunningAvgSamplesPerSec=5.877192567504644, CurrSamplesPerSec=5.304330461192507, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3390/5000 [11:41:04<4:55:29, 11.01s/it][2022-12-20 22:03:39,513] [INFO] [timer.py:197:stop] 0/6864, RunningAvgSamplesPerSec=5.877185822870018, CurrSamplesPerSec=5.296505462651388, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3391/5000 [11:41:15<4:55:39, 11.03s/it][2022-12-20 22:03:50,546] [INFO] [timer.py:197:stop] 0/6866, RunningAvgSamplesPerSec=5.877182587108042, CurrSamplesPerSec=5.315120586018002, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3392/5000 [11:41:26<4:55:32, 11.03s/it][2022-12-20 22:04:01,520] [INFO] [timer.py:197:stop] 0/6868, RunningAvgSamplesPerSec=5.877188798932847, CurrSamplesPerSec=5.365697890343939, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3393/5000 [11:41:37<4:54:55, 11.01s/it][2022-12-20 22:04:12,526] [INFO] [timer.py:197:stop] 0/6870, RunningAvgSamplesPerSec=5.877189667819138, CurrSamplesPerSec=5.317951822471385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3394/5000 [11:41:48<4:54:41, 11.01s/it][2022-12-20 22:04:23,575] [INFO] [timer.py:197:stop] 0/6872, RunningAvgSamplesPerSec=5.877183964858576, CurrSamplesPerSec=5.30589748114639, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3395/5000 [11:41:59<4:54:49, 11.02s/it][2022-12-20 22:04:34,604] [INFO] [timer.py:197:stop] 0/6874, RunningAvgSamplesPerSec=5.8771813348065285, CurrSamplesPerSec=5.322737487954535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3396/5000 [11:42:10<4:54:42, 11.02s/it][2022-12-20 22:04:45,637] [INFO] [timer.py:197:stop] 0/6876, RunningAvgSamplesPerSec=5.877178199969104, CurrSamplesPerSec=5.3268259918107095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3397/5000 [11:42:21<4:54:35, 11.03s/it][2022-12-20 22:04:56,675] [INFO] [timer.py:197:stop] 0/6878, RunningAvgSamplesPerSec=5.877174091521779, CurrSamplesPerSec=5.315441591488052, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3398/5000 [11:42:32<4:54:29, 11.03s/it][2022-12-20 22:05:07,623] [INFO] [logging.py:68:log_dist] [Rank 0] step=3440, skipped=5, lr=[3.48e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:05:07,625] [INFO] [timer.py:197:stop] 0/6880, RunningAvgSamplesPerSec=5.877183610872277, CurrSamplesPerSec=5.354042897372087, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3399/5000 [11:42:43<4:53:40, 11.01s/it][2022-12-20 22:05:18,695] [INFO] [timer.py:197:stop] 0/6882, RunningAvgSamplesPerSec=5.877174669428006, CurrSamplesPerSec=5.305327434208992, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3400/5000 [11:42:54<4:54:00, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.4777777777777784e-06, 'epoch': 82.92}

 68%|██████▊   | 3400/5000 [11:42:54<4:54:00, 11.03s/it][2022-12-20 22:05:29,703] [INFO] [timer.py:197:stop] 0/6884, RunningAvgSamplesPerSec=5.877175480106373, CurrSamplesPerSec=5.301469521761593, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3401/5000 [11:43:05<4:53:41, 11.02s/it][2022-12-20 22:05:40,777] [INFO] [timer.py:197:stop] 0/6886, RunningAvgSamplesPerSec=5.877165702040161, CurrSamplesPerSec=5.2787243944397915, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3402/5000 [11:43:16<4:53:55, 11.04s/it][2022-12-20 22:05:51,759] [INFO] [timer.py:197:stop] 0/6888, RunningAvgSamplesPerSec=5.877170358936834, CurrSamplesPerSec=5.3579554890153265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3403/5000 [11:43:27<4:53:18, 11.02s/it][2022-12-20 22:06:01,833] [INFO] [timer.py:197:stop] 0/6890, RunningAvgSamplesPerSec=5.877315810523471, CurrSamplesPerSec=5.340438759763765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3404/5000 [11:43:42<5:24:45, 12.21s/it][2022-12-20 22:06:12,807] [INFO] [timer.py:197:stop] 0/6892, RunningAvgSamplesPerSec=5.8773216716347445, CurrSamplesPerSec=5.331350216492948, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3405/5000 [11:43:53<5:14:32, 11.83s/it][2022-12-20 22:06:23,739] [INFO] [timer.py:197:stop] 0/6894, RunningAvgSamplesPerSec=5.877334025669836, CurrSamplesPerSec=5.351716990418064, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3406/5000 [11:44:04<5:07:17, 11.57s/it][2022-12-20 22:06:34,752] [INFO] [timer.py:197:stop] 0/6896, RunningAvgSamplesPerSec=5.877334027925976, CurrSamplesPerSec=5.2937738454589445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3407/5000 [11:44:15<5:03:02, 11.41s/it][2022-12-20 22:06:45,781] [INFO] [timer.py:197:stop] 0/6898, RunningAvgSamplesPerSec=5.877331708515138, CurrSamplesPerSec=5.318920829681682, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3408/5000 [11:44:26<4:59:37, 11.29s/it][2022-12-20 22:06:56,800] [INFO] [logging.py:68:log_dist] [Rank 0] step=3450, skipped=5, lr=[3.457777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:06:56,802] [INFO] [timer.py:197:stop] 0/6900, RunningAvgSamplesPerSec=5.877330466214841, CurrSamplesPerSec=5.30837770866021, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3409/5000 [11:44:37<4:57:40, 11.23s/it][2022-12-20 22:07:07,874] [INFO] [timer.py:197:stop] 0/6902, RunningAvgSamplesPerSec=5.877321256762561, CurrSamplesPerSec=5.30653059071755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3410/5000 [11:44:48<4:56:01, 11.17s/it][2022-12-20 22:07:18,918] [INFO] [timer.py:197:stop] 0/6904, RunningAvgSamplesPerSec=5.87731650635782, CurrSamplesPerSec=5.30544738976029, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3411/5000 [11:44:59<4:54:59, 11.14s/it][2022-12-20 22:07:29,988] [INFO] [timer.py:197:stop] 0/6906, RunningAvgSamplesPerSec=5.877307861837475, CurrSamplesPerSec=5.3009561156185665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3412/5000 [11:45:10<4:54:15, 11.12s/it][2022-12-20 22:07:41,006] [INFO] [timer.py:197:stop] 0/6908, RunningAvgSamplesPerSec=5.8773070544836505, CurrSamplesPerSec=5.345843237802084, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3413/5000 [11:45:21<4:52:45, 11.07s/it][2022-12-20 22:07:51,986] [INFO] [timer.py:197:stop] 0/6910, RunningAvgSamplesPerSec=5.8773121307858425, CurrSamplesPerSec=5.320537610935107, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3414/5000 [11:45:32<4:51:57, 11.04s/it][2022-12-20 22:08:02,989] [INFO] [timer.py:197:stop] 0/6912, RunningAvgSamplesPerSec=5.877313812463177, CurrSamplesPerSec=5.309628039266594, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3415/5000 [11:45:43<4:51:55, 11.05s/it][2022-12-20 22:08:14,050] [INFO] [timer.py:197:stop] 0/6914, RunningAvgSamplesPerSec=5.877307545383985, CurrSamplesPerSec=5.31239307846239, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3416/5000 [11:45:54<4:51:24, 11.04s/it][2022-12-20 22:08:25,006] [INFO] [timer.py:197:stop] 0/6916, RunningAvgSamplesPerSec=5.877316099520914, CurrSamplesPerSec=5.359670785574039, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3417/5000 [11:46:05<4:50:43, 11.02s/it][2022-12-20 22:08:35,982] [INFO] [timer.py:197:stop] 0/6918, RunningAvgSamplesPerSec=5.877321686583255, CurrSamplesPerSec=5.357892606406187, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3418/5000 [11:46:16<4:50:22, 11.01s/it][2022-12-20 22:08:46,988] [INFO] [logging.py:68:log_dist] [Rank 0] step=3460, skipped=5, lr=[3.435555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:08:46,990] [INFO] [timer.py:197:stop] 0/6920, RunningAvgSamplesPerSec=5.87732227539434, CurrSamplesPerSec=5.349147714739017, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3419/5000 [11:46:27<4:50:07, 11.01s/it][2022-12-20 22:08:57,993] [INFO] [timer.py:197:stop] 0/6922, RunningAvgSamplesPerSec=5.877323720942297, CurrSamplesPerSec=5.350932039540956, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3420/5000 [11:46:38<4:49:33, 11.00s/it][2022-12-20 22:09:08,973] [INFO] [timer.py:197:stop] 0/6924, RunningAvgSamplesPerSec=5.877328750587015, CurrSamplesPerSec=5.3352971417675095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3421/5000 [11:46:49<4:49:25, 11.00s/it][2022-12-20 22:09:19,940] [INFO] [timer.py:197:stop] 0/6926, RunningAvgSamplesPerSec=5.877335502436876, CurrSamplesPerSec=5.365227516559954, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3422/5000 [11:47:00<4:49:04, 10.99s/it][2022-12-20 22:09:30,940] [INFO] [timer.py:197:stop] 0/6928, RunningAvgSamplesPerSec=5.877337508857378, CurrSamplesPerSec=5.345756579550044, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3423/5000 [11:47:11<4:48:58, 10.99s/it][2022-12-20 22:09:41,940] [INFO] [timer.py:197:stop] 0/6930, RunningAvgSamplesPerSec=5.877339467387723, CurrSamplesPerSec=5.348187056626314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3424/5000 [11:47:22<4:48:32, 10.99s/it][2022-12-20 22:09:52,938] [INFO] [timer.py:197:stop] 0/6932, RunningAvgSamplesPerSec=5.8773416822668, CurrSamplesPerSec=5.316915978074382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 68%|██████▊   | 3425/5000 [11:47:33<4:48:39, 11.00s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.4222222222222224e-06, 'epoch': 83.53}

 68%|██████▊   | 3425/5000 [11:47:33<4:48:39, 11.00s/it][2022-12-20 22:10:03,960] [INFO] [timer.py:197:stop] 0/6934, RunningAvgSamplesPerSec=5.877340372004912, CurrSamplesPerSec=5.318010188944311, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3426/5000 [11:47:44<4:48:37, 11.00s/it][2022-12-20 22:10:14,971] [INFO] [timer.py:197:stop] 0/6936, RunningAvgSamplesPerSec=5.87734058442746, CurrSamplesPerSec=5.3208337477710055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3427/5000 [11:47:55<4:48:37, 11.01s/it][2022-12-20 22:10:25,949] [INFO] [timer.py:197:stop] 0/6938, RunningAvgSamplesPerSec=5.8773461176789725, CurrSamplesPerSec=5.363545527449814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3428/5000 [11:48:06<4:47:50, 10.99s/it][2022-12-20 22:10:36,915] [INFO] [logging.py:68:log_dist] [Rank 0] step=3470, skipped=5, lr=[3.4133333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:10:36,916] [INFO] [timer.py:197:stop] 0/6940, RunningAvgSamplesPerSec=5.877353156269629, CurrSamplesPerSec=5.331998311624465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3429/5000 [11:48:17<4:47:56, 11.00s/it][2022-12-20 22:10:47,939] [INFO] [timer.py:197:stop] 0/6942, RunningAvgSamplesPerSec=5.877351457895751, CurrSamplesPerSec=5.3316217195243825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3430/5000 [11:48:28<4:48:08, 11.01s/it][2022-12-20 22:10:58,979] [INFO] [timer.py:197:stop] 0/6944, RunningAvgSamplesPerSec=5.877347136855028, CurrSamplesPerSec=5.338147372190629, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3431/5000 [11:48:39<4:47:15, 10.99s/it][2022-12-20 22:11:09,910] [INFO] [timer.py:197:stop] 0/6946, RunningAvgSamplesPerSec=5.877359823604158, CurrSamplesPerSec=5.3311636536065645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3432/5000 [11:48:50<4:46:58, 10.98s/it][2022-12-20 22:11:20,881] [INFO] [timer.py:197:stop] 0/6948, RunningAvgSamplesPerSec=5.8773660226005235, CurrSamplesPerSec=5.331086999315391, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3433/5000 [11:49:01<4:47:04, 10.99s/it][2022-12-20 22:11:31,922] [INFO] [timer.py:197:stop] 0/6950, RunningAvgSamplesPerSec=5.8773615738378995, CurrSamplesPerSec=5.309442783495795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3434/5000 [11:49:12<4:47:22, 11.01s/it][2022-12-20 22:11:42,963] [INFO] [timer.py:197:stop] 0/6952, RunningAvgSamplesPerSec=5.877357190817917, CurrSamplesPerSec=5.321452493278688, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3435/5000 [11:49:23<4:47:24, 11.02s/it][2022-12-20 22:11:53,985] [INFO] [timer.py:197:stop] 0/6954, RunningAvgSamplesPerSec=5.877355543265012, CurrSamplesPerSec=5.33426915797919, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3436/5000 [11:49:34<4:47:14, 11.02s/it][2022-12-20 22:12:04,978] [INFO] [timer.py:197:stop] 0/6956, RunningAvgSamplesPerSec=5.877358286494414, CurrSamplesPerSec=5.359462546457848, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▊   | 3437/5000 [11:49:45<4:46:37, 11.00s/it][2022-12-20 22:12:15,980] [INFO] [timer.py:197:stop] 0/6958, RunningAvgSamplesPerSec=5.877359875690221, CurrSamplesPerSec=5.326138784018202, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3438/5000 [11:49:56<4:46:33, 11.01s/it][2022-12-20 22:12:26,975] [INFO] [logging.py:68:log_dist] [Rank 0] step=3480, skipped=5, lr=[3.391111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:12:26,976] [INFO] [timer.py:197:stop] 0/6960, RunningAvgSamplesPerSec=5.877362221208545, CurrSamplesPerSec=5.345126209359258, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3439/5000 [11:50:07<4:46:18, 11.00s/it][2022-12-20 22:12:38,017] [INFO] [timer.py:197:stop] 0/6962, RunningAvgSamplesPerSec=5.8773576711673305, CurrSamplesPerSec=5.308471137596698, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3440/5000 [11:50:18<4:46:24, 11.02s/it][2022-12-20 22:12:49,061] [INFO] [timer.py:197:stop] 0/6964, RunningAvgSamplesPerSec=5.877352550160243, CurrSamplesPerSec=5.304979342166683, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3441/5000 [11:50:29<4:46:17, 11.02s/it][2022-12-20 22:13:00,036] [INFO] [timer.py:197:stop] 0/6966, RunningAvgSamplesPerSec=5.8773581789319955, CurrSamplesPerSec=5.3496449114079105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3442/5000 [11:50:40<4:45:54, 11.01s/it][2022-12-20 22:13:11,036] [INFO] [timer.py:197:stop] 0/6968, RunningAvgSamplesPerSec=5.8773600034051725, CurrSamplesPerSec=5.343082201697642, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3443/5000 [11:50:51<4:45:42, 11.01s/it][2022-12-20 22:13:22,079] [INFO] [timer.py:197:stop] 0/6970, RunningAvgSamplesPerSec=5.8773551879723875, CurrSamplesPerSec=5.313720403230395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3444/5000 [11:51:02<4:45:44, 11.02s/it][2022-12-20 22:13:32,205] [INFO] [timer.py:197:stop] 0/6972, RunningAvgSamplesPerSec=5.877490869911894, CurrSamplesPerSec=6.248447254125932, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 22:13:43,165] [INFO] [timer.py:197:stop] 0/6974, RunningAvgSamplesPerSec=5.877498574830126, CurrSamplesPerSec=5.322725244972642, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3445/5000 [11:51:18<5:25:20, 12.55s/it][2022-12-20 22:13:54,209] [INFO] [timer.py:197:stop] 0/6976, RunningAvgSamplesPerSec=5.877493587797274, CurrSamplesPerSec=5.316052766776635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3446/5000 [11:51:29<5:13:24, 12.10s/it][2022-12-20 22:14:05,171] [INFO] [timer.py:197:stop] 0/6978, RunningAvgSamplesPerSec=5.877501197279952, CurrSamplesPerSec=5.343533808555723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3447/5000 [11:51:40<5:04:21, 11.76s/it][2022-12-20 22:14:16,242] [INFO] [logging.py:68:log_dist] [Rank 0] step=3490, skipped=5, lr=[3.3688888888888895e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:14:16,243] [INFO] [timer.py:197:stop] 0/6980, RunningAvgSamplesPerSec=5.877491669252841, CurrSamplesPerSec=5.30939153588281, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3448/5000 [11:51:51<4:58:50, 11.55s/it][2022-12-20 22:14:27,293] [INFO] [timer.py:197:stop] 0/6982, RunningAvgSamplesPerSec=5.877485675721347, CurrSamplesPerSec=5.305356583747886, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3449/5000 [11:52:02<4:54:44, 11.40s/it][2022-12-20 22:14:38,330] [INFO] [timer.py:197:stop] 0/6984, RunningAvgSamplesPerSec=5.87748152849938, CurrSamplesPerSec=5.2929792895173025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3450/5000 [11:52:13<4:51:43, 11.29s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.3644444444444447e-06, 'epoch': 84.14}

 69%|██████▉   | 3450/5000 [11:52:13<4:51:43, 11.29s/it][2022-12-20 22:14:49,349] [INFO] [timer.py:197:stop] 0/6986, RunningAvgSamplesPerSec=5.8774805628949025, CurrSamplesPerSec=5.348528906494639, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3451/5000 [11:52:24<4:49:24, 11.21s/it][2022-12-20 22:15:00,353] [INFO] [timer.py:197:stop] 0/6988, RunningAvgSamplesPerSec=5.877481684872852, CurrSamplesPerSec=5.328756448799737, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3452/5000 [11:52:35<4:47:38, 11.15s/it][2022-12-20 22:15:11,421] [INFO] [timer.py:197:stop] 0/6990, RunningAvgSamplesPerSec=5.8774728428873955, CurrSamplesPerSec=5.29729208753428, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3453/5000 [11:52:47<4:46:49, 11.12s/it][2022-12-20 22:15:22,439] [INFO] [timer.py:197:stop] 0/6992, RunningAvgSamplesPerSec=5.877471801103504, CurrSamplesPerSec=5.31107965685935, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3454/5000 [11:52:58<4:45:48, 11.09s/it][2022-12-20 22:15:33,513] [INFO] [timer.py:197:stop] 0/6994, RunningAvgSamplesPerSec=5.8774620629374, CurrSamplesPerSec=5.295767965837891, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3455/5000 [11:53:09<4:45:29, 11.09s/it][2022-12-20 22:15:44,588] [INFO] [timer.py:197:stop] 0/6996, RunningAvgSamplesPerSec=5.8774523689021265, CurrSamplesPerSec=5.280769313544631, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3456/5000 [11:53:20<4:45:12, 11.08s/it][2022-12-20 22:15:55,610] [INFO] [timer.py:197:stop] 0/6998, RunningAvgSamplesPerSec=5.877450942068787, CurrSamplesPerSec=5.349364747201171, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3457/5000 [11:53:31<4:44:33, 11.06s/it][2022-12-20 22:16:06,647] [INFO] [logging.py:68:log_dist] [Rank 0] step=3500, skipped=5, lr=[3.346666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:16:06,648] [INFO] [timer.py:197:stop] 0/7000, RunningAvgSamplesPerSec=5.877446973922996, CurrSamplesPerSec=5.331681445430516, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3458/5000 [11:53:42<4:44:09, 11.06s/it][2022-12-20 22:16:17,675] [INFO] [timer.py:197:stop] 0/7002, RunningAvgSamplesPerSec=5.87744472253816, CurrSamplesPerSec=5.341243378300105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3459/5000 [11:53:53<4:43:44, 11.05s/it][2022-12-20 22:16:28,708] [INFO] [timer.py:197:stop] 0/7004, RunningAvgSamplesPerSec=5.8774416771214275, CurrSamplesPerSec=5.323021414280375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3460/5000 [11:54:04<4:43:26, 11.04s/it][2022-12-20 22:16:39,693] [INFO] [timer.py:197:stop] 0/7006, RunningAvgSamplesPerSec=5.8774459871268565, CurrSamplesPerSec=5.32867055518247, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3461/5000 [11:54:15<4:42:48, 11.03s/it][2022-12-20 22:16:50,746] [INFO] [timer.py:197:stop] 0/7008, RunningAvgSamplesPerSec=5.877439716528593, CurrSamplesPerSec=5.32086791952723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3462/5000 [11:54:26<4:42:50, 11.03s/it][2022-12-20 22:17:01,761] [INFO] [timer.py:197:stop] 0/7010, RunningAvgSamplesPerSec=5.877439540836382, CurrSamplesPerSec=5.312082742927017, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3463/5000 [11:54:37<4:42:30, 11.03s/it][2022-12-20 22:17:12,775] [INFO] [timer.py:197:stop] 0/7012, RunningAvgSamplesPerSec=5.877439543277474, CurrSamplesPerSec=5.338243338209372, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3464/5000 [11:54:48<4:42:12, 11.02s/it][2022-12-20 22:17:23,779] [INFO] [timer.py:197:stop] 0/7014, RunningAvgSamplesPerSec=5.877440924391854, CurrSamplesPerSec=5.342185808522981, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3465/5000 [11:54:59<4:41:52, 11.02s/it][2022-12-20 22:17:34,768] [INFO] [timer.py:197:stop] 0/7016, RunningAvgSamplesPerSec=5.877444642637558, CurrSamplesPerSec=5.362222760859554, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3466/5000 [11:55:10<4:41:28, 11.01s/it][2022-12-20 22:17:45,774] [INFO] [timer.py:197:stop] 0/7018, RunningAvgSamplesPerSec=5.877445710505525, CurrSamplesPerSec=5.320061625288502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3467/5000 [11:55:21<4:41:15, 11.01s/it][2022-12-20 22:17:56,775] [INFO] [logging.py:68:log_dist] [Rank 0] step=3510, skipped=5, lr=[3.3244444444444447e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:17:56,777] [INFO] [timer.py:197:stop] 0/7020, RunningAvgSamplesPerSec=5.877447211724594, CurrSamplesPerSec=5.354907809005693, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3468/5000 [11:55:32<4:41:02, 11.01s/it][2022-12-20 22:18:07,799] [INFO] [timer.py:197:stop] 0/7022, RunningAvgSamplesPerSec=5.877445783028536, CurrSamplesPerSec=5.320336619455907, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3469/5000 [11:55:43<4:40:58, 11.01s/it][2022-12-20 22:18:18,738] [INFO] [timer.py:197:stop] 0/7024, RunningAvgSamplesPerSec=5.877457066854889, CurrSamplesPerSec=5.363328628719668, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3470/5000 [11:55:54<4:40:13, 10.99s/it][2022-12-20 22:18:29,758] [INFO] [timer.py:197:stop] 0/7026, RunningAvgSamplesPerSec=5.8774561109742365, CurrSamplesPerSec=5.331781203385353, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3471/5000 [11:56:05<4:40:16, 11.00s/it][2022-12-20 22:18:40,769] [INFO] [timer.py:197:stop] 0/7028, RunningAvgSamplesPerSec=5.877456183605429, CurrSamplesPerSec=5.313672228525821, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3472/5000 [11:56:16<4:40:11, 11.00s/it][2022-12-20 22:18:51,767] [INFO] [timer.py:197:stop] 0/7030, RunningAvgSamplesPerSec=5.87745845528586, CurrSamplesPerSec=5.356988244001654, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3473/5000 [11:56:27<4:39:58, 11.00s/it][2022-12-20 22:19:02,728] [INFO] [timer.py:197:stop] 0/7032, RunningAvgSamplesPerSec=5.877466353741475, CurrSamplesPerSec=5.391105280846113, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 69%|██████▉   | 3474/5000 [11:56:38<4:39:29, 10.99s/it][2022-12-20 22:19:13,681] [INFO] [timer.py:197:stop] 0/7034, RunningAvgSamplesPerSec=5.877475532310344, CurrSamplesPerSec=5.320628937369722, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3475/5000 [11:56:49<4:39:01, 10.98s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.308888888888889e-06, 'epoch': 84.75}

 70%|██████▉   | 3475/5000 [11:56:49<4:39:01, 10.98s/it][2022-12-20 22:19:24,749] [INFO] [timer.py:197:stop] 0/7036, RunningAvgSamplesPerSec=5.8774672265804115, CurrSamplesPerSec=5.312515666783656, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3476/5000 [11:57:00<4:39:31, 11.01s/it][2022-12-20 22:19:35,715] [INFO] [timer.py:197:stop] 0/7038, RunningAvgSamplesPerSec=5.877474541192782, CurrSamplesPerSec=5.351275520558105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3477/5000 [11:57:11<4:39:03, 10.99s/it][2022-12-20 22:19:46,728] [INFO] [logging.py:68:log_dist] [Rank 0] step=3520, skipped=5, lr=[3.3022222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:19:46,729] [INFO] [timer.py:197:stop] 0/7040, RunningAvgSamplesPerSec=5.877474247885195, CurrSamplesPerSec=5.321180970116635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3478/5000 [11:57:22<4:39:01, 11.00s/it][2022-12-20 22:19:57,672] [INFO] [timer.py:197:stop] 0/7042, RunningAvgSamplesPerSec=5.8774851341137335, CurrSamplesPerSec=5.354193900336188, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3479/5000 [11:57:33<4:38:24, 10.98s/it][2022-12-20 22:20:08,630] [INFO] [timer.py:197:stop] 0/7044, RunningAvgSamplesPerSec=5.877493624973739, CurrSamplesPerSec=5.375719057651177, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3480/5000 [11:57:44<4:38:02, 10.98s/it][2022-12-20 22:20:19,636] [INFO] [timer.py:197:stop] 0/7046, RunningAvgSamplesPerSec=5.877494576234701, CurrSamplesPerSec=5.328568163453796, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3481/5000 [11:57:55<4:38:05, 10.98s/it][2022-12-20 22:20:30,624] [INFO] [timer.py:197:stop] 0/7048, RunningAvgSamplesPerSec=5.8774982361315065, CurrSamplesPerSec=5.324680402686268, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3482/5000 [11:58:06<4:37:56, 10.99s/it][2022-12-20 22:20:41,571] [INFO] [timer.py:197:stop] 0/7050, RunningAvgSamplesPerSec=5.877508339212149, CurrSamplesPerSec=5.36354767080543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3483/5000 [11:58:17<4:37:27, 10.97s/it][2022-12-20 22:20:52,577] [INFO] [timer.py:197:stop] 0/7052, RunningAvgSamplesPerSec=5.8775094210481065, CurrSamplesPerSec=5.316602376096449, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3484/5000 [11:58:28<4:37:31, 10.98s/it][2022-12-20 22:21:03,559] [INFO] [timer.py:197:stop] 0/7054, RunningAvgSamplesPerSec=5.877514208826281, CurrSamplesPerSec=5.3564197789727075, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3485/5000 [11:58:39<4:37:19, 10.98s/it][2022-12-20 22:21:13,628] [INFO] [timer.py:197:stop] 0/7056, RunningAvgSamplesPerSec=5.8776571352838305, CurrSamplesPerSec=5.339012889626352, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3486/5000 [11:58:54<5:07:35, 12.19s/it][2022-12-20 22:21:24,651] [INFO] [timer.py:197:stop] 0/7058, RunningAvgSamplesPerSec=5.877655314727172, CurrSamplesPerSec=5.312251362479783, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3487/5000 [11:59:05<4:58:35, 11.84s/it][2022-12-20 22:21:35,672] [INFO] [logging.py:68:log_dist] [Rank 0] step=3530, skipped=5, lr=[3.2800000000000004e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:21:35,674] [INFO] [timer.py:197:stop] 0/7060, RunningAvgSamplesPerSec=5.877653715399471, CurrSamplesPerSec=5.317598911708139, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3488/5000 [11:59:16<4:51:54, 11.58s/it][2022-12-20 22:21:46,646] [INFO] [timer.py:197:stop] 0/7062, RunningAvgSamplesPerSec=5.87765967397203, CurrSamplesPerSec=5.326693651947659, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3489/5000 [11:59:27<4:47:08, 11.40s/it][2022-12-20 22:21:57,612] [INFO] [timer.py:197:stop] 0/7064, RunningAvgSamplesPerSec=5.877666521024726, CurrSamplesPerSec=5.3363323111485075, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3490/5000 [11:59:38<4:43:37, 11.27s/it][2022-12-20 22:22:08,581] [INFO] [timer.py:197:stop] 0/7066, RunningAvgSamplesPerSec=5.877672971855654, CurrSamplesPerSec=5.3313442869381085, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3491/5000 [11:59:49<4:41:25, 11.19s/it][2022-12-20 22:22:19,605] [INFO] [timer.py:197:stop] 0/7068, RunningAvgSamplesPerSec=5.877671065308999, CurrSamplesPerSec=5.312624171630035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3492/5000 [12:00:00<4:40:02, 11.14s/it][2022-12-20 22:22:30,605] [INFO] [timer.py:197:stop] 0/7070, RunningAvgSamplesPerSec=5.8776729225194995, CurrSamplesPerSec=5.341195128359186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3493/5000 [12:00:11<4:38:37, 11.09s/it][2022-12-20 22:22:41,606] [INFO] [timer.py:197:stop] 0/7072, RunningAvgSamplesPerSec=5.877674629740443, CurrSamplesPerSec=5.321400380624145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3494/5000 [12:00:22<4:37:59, 11.08s/it][2022-12-20 22:22:52,613] [INFO] [timer.py:197:stop] 0/7074, RunningAvgSamplesPerSec=5.877675350635434, CurrSamplesPerSec=5.344193168259925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3495/5000 [12:00:33<4:37:11, 11.05s/it][2022-12-20 22:23:03,564] [INFO] [timer.py:197:stop] 0/7076, RunningAvgSamplesPerSec=5.877684793483298, CurrSamplesPerSec=5.383221423642713, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3496/5000 [12:00:44<4:36:00, 11.01s/it][2022-12-20 22:23:14,495] [INFO] [timer.py:197:stop] 0/7078, RunningAvgSamplesPerSec=5.877697172324611, CurrSamplesPerSec=5.3711794328447535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3497/5000 [12:00:55<4:35:28, 11.00s/it][2022-12-20 22:23:25,472] [INFO] [logging.py:68:log_dist] [Rank 0] step=3540, skipped=5, lr=[3.257777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:23:25,473] [INFO] [timer.py:197:stop] 0/7080, RunningAvgSamplesPerSec=5.877702089924552, CurrSamplesPerSec=5.358249388533631, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3498/5000 [12:01:06<4:35:11, 10.99s/it][2022-12-20 22:23:36,526] [INFO] [timer.py:197:stop] 0/7082, RunningAvgSamplesPerSec=5.877696122526558, CurrSamplesPerSec=5.2985487052325855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|██████▉   | 3499/5000 [12:01:17<4:35:29, 11.01s/it][2022-12-20 22:23:47,579] [INFO] [timer.py:197:stop] 0/7084, RunningAvgSamplesPerSec=5.877689998628411, CurrSamplesPerSec=5.302050259562705, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3500/5000 [12:01:28<4:35:41, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.2533333333333332e-06, 'epoch': 85.36}

 70%|███████   | 3500/5000 [12:01:28<4:35:41, 11.03s/it][2022-12-20 22:23:58,603] [INFO] [timer.py:197:stop] 0/7086, RunningAvgSamplesPerSec=5.877688186615257, CurrSamplesPerSec=5.335869405029359, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3501/5000 [12:01:39<4:35:29, 11.03s/it][2022-12-20 22:24:09,642] [INFO] [timer.py:197:stop] 0/7088, RunningAvgSamplesPerSec=5.877683984337105, CurrSamplesPerSec=5.323403548821259, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3502/5000 [12:01:50<4:35:15, 11.02s/it][2022-12-20 22:24:20,681] [INFO] [timer.py:197:stop] 0/7090, RunningAvgSamplesPerSec=5.8776796667779605, CurrSamplesPerSec=5.306378068402197, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3503/5000 [12:02:01<4:35:24, 11.04s/it][2022-12-20 22:24:31,708] [INFO] [timer.py:197:stop] 0/7092, RunningAvgSamplesPerSec=5.8776771634151626, CurrSamplesPerSec=5.344237003682219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3504/5000 [12:02:12<4:34:56, 11.03s/it][2022-12-20 22:24:42,738] [INFO] [timer.py:197:stop] 0/7094, RunningAvgSamplesPerSec=5.877674349049392, CurrSamplesPerSec=5.318041163730092, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3505/5000 [12:02:23<4:34:19, 11.01s/it][2022-12-20 22:24:53,727] [INFO] [timer.py:197:stop] 0/7096, RunningAvgSamplesPerSec=5.877677810319436, CurrSamplesPerSec=5.301913911925116, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3506/5000 [12:02:34<4:34:29, 11.02s/it][2022-12-20 22:25:04,790] [INFO] [timer.py:197:stop] 0/7098, RunningAvgSamplesPerSec=5.877669958957125, CurrSamplesPerSec=5.296598683160091, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3507/5000 [12:02:45<4:34:13, 11.02s/it][2022-12-20 22:25:15,780] [INFO] [logging.py:68:log_dist] [Rank 0] step=3550, skipped=5, lr=[3.2355555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:25:15,782] [INFO] [timer.py:197:stop] 0/7100, RunningAvgSamplesPerSec=5.8776727394301815, CurrSamplesPerSec=5.314050917879063, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3508/5000 [12:02:56<4:34:00, 11.02s/it][2022-12-20 22:25:26,803] [INFO] [timer.py:197:stop] 0/7102, RunningAvgSamplesPerSec=5.877671050061761, CurrSamplesPerSec=5.309400987212443, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3509/5000 [12:03:07<4:33:40, 11.01s/it][2022-12-20 22:25:37,745] [INFO] [timer.py:197:stop] 0/7104, RunningAvgSamplesPerSec=5.8776817427717685, CurrSamplesPerSec=5.361654897883952, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3510/5000 [12:03:18<4:33:16, 11.00s/it][2022-12-20 22:25:48,787] [INFO] [timer.py:197:stop] 0/7106, RunningAvgSamplesPerSec=5.877677144189214, CurrSamplesPerSec=5.309730124501472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3511/5000 [12:03:29<4:33:20, 11.01s/it][2022-12-20 22:25:59,782] [INFO] [timer.py:197:stop] 0/7108, RunningAvgSamplesPerSec=5.877679893653706, CurrSamplesPerSec=5.348395486072259, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3512/5000 [12:03:40<4:33:03, 11.01s/it][2022-12-20 22:26:10,827] [INFO] [timer.py:197:stop] 0/7110, RunningAvgSamplesPerSec=5.877674698799316, CurrSamplesPerSec=5.307483056951617, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3513/5000 [12:03:51<4:33:15, 11.03s/it][2022-12-20 22:26:21,871] [INFO] [timer.py:197:stop] 0/7112, RunningAvgSamplesPerSec=5.877669801342578, CurrSamplesPerSec=5.324243170735287, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3514/5000 [12:04:02<4:32:54, 11.02s/it][2022-12-20 22:26:32,889] [INFO] [timer.py:197:stop] 0/7114, RunningAvgSamplesPerSec=5.877668691859985, CurrSamplesPerSec=5.310946206864896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3515/5000 [12:04:13<4:32:58, 11.03s/it][2022-12-20 22:26:44,030] [INFO] [timer.py:197:stop] 0/7116, RunningAvgSamplesPerSec=5.877648971453292, CurrSamplesPerSec=5.234629936950541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3516/5000 [12:04:24<4:33:34, 11.06s/it][2022-12-20 22:26:55,070] [INFO] [timer.py:197:stop] 0/7118, RunningAvgSamplesPerSec=5.877644675927699, CurrSamplesPerSec=5.317994385621187, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3517/5000 [12:04:35<4:33:10, 11.05s/it][2022-12-20 22:27:06,131] [INFO] [logging.py:68:log_dist] [Rank 0] step=3560, skipped=5, lr=[3.213333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:27:06,133] [INFO] [timer.py:197:stop] 0/7120, RunningAvgSamplesPerSec=5.877636627018262, CurrSamplesPerSec=5.289335031214186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3518/5000 [12:04:46<4:32:54, 11.05s/it][2022-12-20 22:27:17,067] [INFO] [timer.py:197:stop] 0/7122, RunningAvgSamplesPerSec=5.877648209443444, CurrSamplesPerSec=5.384742945985685, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3519/5000 [12:04:57<4:31:43, 11.01s/it][2022-12-20 22:27:28,054] [INFO] [timer.py:197:stop] 0/7124, RunningAvgSamplesPerSec=5.8776519509022656, CurrSamplesPerSec=5.319924138864816, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3520/5000 [12:05:08<4:31:48, 11.02s/it][2022-12-20 22:27:39,060] [INFO] [timer.py:197:stop] 0/7126, RunningAvgSamplesPerSec=5.8776526506763265, CurrSamplesPerSec=5.353181894723441, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3521/5000 [12:05:19<4:31:23, 11.01s/it][2022-12-20 22:27:50,074] [INFO] [timer.py:197:stop] 0/7128, RunningAvgSamplesPerSec=5.877652281075482, CurrSamplesPerSec=5.3304933208447345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3522/5000 [12:05:30<4:31:03, 11.00s/it][2022-12-20 22:28:01,069] [INFO] [timer.py:197:stop] 0/7130, RunningAvgSamplesPerSec=5.87765492605659, CurrSamplesPerSec=5.327145664099563, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3523/5000 [12:05:41<4:30:48, 11.00s/it][2022-12-20 22:28:12,055] [INFO] [timer.py:197:stop] 0/7132, RunningAvgSamplesPerSec=5.877658583682138, CurrSamplesPerSec=5.3326690229818405, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3524/5000 [12:05:52<4:30:50, 11.01s/it][2022-12-20 22:28:23,076] [INFO] [timer.py:197:stop] 0/7134, RunningAvgSamplesPerSec=5.87765711447172, CurrSamplesPerSec=5.340683350052266, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 70%|███████   | 3525/5000 [12:06:03<4:30:46, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.197777777777778e-06, 'epoch': 85.96}

 70%|███████   | 3525/5000 [12:06:03<4:30:46, 11.01s/it][2022-12-20 22:28:34,127] [INFO] [timer.py:197:stop] 0/7136, RunningAvgSamplesPerSec=5.877651390789766, CurrSamplesPerSec=5.3199833921116495, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3526/5000 [12:06:14<4:30:39, 11.02s/it][2022-12-20 22:28:44,233] [INFO] [timer.py:197:stop] 0/7138, RunningAvgSamplesPerSec=5.877786769108438, CurrSamplesPerSec=6.264182824637949, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 22:28:55,270] [INFO] [logging.py:68:log_dist] [Rank 0] step=3570, skipped=5, lr=[3.1911111111111117e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:28:55,272] [INFO] [timer.py:197:stop] 0/7140, RunningAvgSamplesPerSec=5.877781944884518, CurrSamplesPerSec=5.3111095001378645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3527/5000 [12:06:30<5:08:39, 12.57s/it][2022-12-20 22:29:06,249] [INFO] [timer.py:197:stop] 0/7142, RunningAvgSamplesPerSec=5.8777870559258085, CurrSamplesPerSec=5.363731577096455, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3528/5000 [12:06:41<4:56:41, 12.09s/it][2022-12-20 22:29:17,233] [INFO] [timer.py:197:stop] 0/7144, RunningAvgSamplesPerSec=5.877791170999546, CurrSamplesPerSec=5.3301186346754585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3529/5000 [12:06:52<4:48:20, 11.76s/it][2022-12-20 22:29:28,278] [INFO] [timer.py:197:stop] 0/7146, RunningAvgSamplesPerSec=5.8777860286597114, CurrSamplesPerSec=5.3143107720748874, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3530/5000 [12:07:03<4:42:52, 11.55s/it][2022-12-20 22:29:39,330] [INFO] [timer.py:197:stop] 0/7148, RunningAvgSamplesPerSec=5.877779888836587, CurrSamplesPerSec=5.310253845666899, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3531/5000 [12:07:14<4:39:03, 11.40s/it][2022-12-20 22:29:50,337] [INFO] [timer.py:197:stop] 0/7150, RunningAvgSamplesPerSec=5.877780560936945, CurrSamplesPerSec=5.341362412719359, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3532/5000 [12:07:25<4:35:59, 11.28s/it][2022-12-20 22:30:01,353] [INFO] [timer.py:197:stop] 0/7152, RunningAvgSamplesPerSec=5.877779704756266, CurrSamplesPerSec=5.329207329924256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3533/5000 [12:07:36<4:33:52, 11.20s/it][2022-12-20 22:30:12,285] [INFO] [timer.py:197:stop] 0/7154, RunningAvgSamplesPerSec=5.877791702046096, CurrSamplesPerSec=5.3291430043470855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3534/5000 [12:07:47<4:31:42, 11.12s/it][2022-12-20 22:30:23,312] [INFO] [timer.py:197:stop] 0/7156, RunningAvgSamplesPerSec=5.8777892056737135, CurrSamplesPerSec=5.312989882426634, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3535/5000 [12:07:58<4:30:50, 11.09s/it][2022-12-20 22:30:34,323] [INFO] [timer.py:197:stop] 0/7158, RunningAvgSamplesPerSec=5.87778925220122, CurrSamplesPerSec=5.311349729728767, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3536/5000 [12:08:09<4:30:03, 11.07s/it][2022-12-20 22:30:45,329] [INFO] [logging.py:68:log_dist] [Rank 0] step=3580, skipped=5, lr=[3.1688888888888893e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:30:45,331] [INFO] [timer.py:197:stop] 0/7160, RunningAvgSamplesPerSec=5.877789568586745, CurrSamplesPerSec=5.299858865472995, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3537/5000 [12:08:20<4:29:26, 11.05s/it][2022-12-20 22:30:56,393] [INFO] [timer.py:197:stop] 0/7162, RunningAvgSamplesPerSec=5.877781900051507, CurrSamplesPerSec=5.304261703787576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3538/5000 [12:08:31<4:29:20, 11.05s/it][2022-12-20 22:31:07,444] [INFO] [timer.py:197:stop] 0/7164, RunningAvgSamplesPerSec=5.877775869269952, CurrSamplesPerSec=5.299717398901105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3539/5000 [12:08:43<4:29:08, 11.05s/it][2022-12-20 22:31:18,437] [INFO] [timer.py:197:stop] 0/7166, RunningAvgSamplesPerSec=5.8777786338945015, CurrSamplesPerSec=5.338908825828695, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3540/5000 [12:08:54<4:28:30, 11.03s/it][2022-12-20 22:31:29,407] [INFO] [timer.py:197:stop] 0/7168, RunningAvgSamplesPerSec=5.877784727835519, CurrSamplesPerSec=5.34269851285209, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3541/5000 [12:09:05<4:27:51, 11.02s/it][2022-12-20 22:31:40,440] [INFO] [timer.py:197:stop] 0/7170, RunningAvgSamplesPerSec=5.877781392208103, CurrSamplesPerSec=5.310631207414719, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3542/5000 [12:09:16<4:27:48, 11.02s/it][2022-12-20 22:31:51,406] [INFO] [timer.py:197:stop] 0/7172, RunningAvgSamplesPerSec=5.877788081665298, CurrSamplesPerSec=5.33450576271111, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3543/5000 [12:09:27<4:27:13, 11.00s/it][2022-12-20 22:32:02,427] [INFO] [timer.py:197:stop] 0/7174, RunningAvgSamplesPerSec=5.8777864796720065, CurrSamplesPerSec=5.338527010154311, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3544/5000 [12:09:38<4:27:09, 11.01s/it][2022-12-20 22:32:13,464] [INFO] [timer.py:197:stop] 0/7176, RunningAvgSamplesPerSec=5.877782595509925, CurrSamplesPerSec=5.285765938053256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3545/5000 [12:09:49<4:27:10, 11.02s/it][2022-12-20 22:32:24,426] [INFO] [timer.py:197:stop] 0/7178, RunningAvgSamplesPerSec=5.87779018756586, CurrSamplesPerSec=5.354629656780461, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3546/5000 [12:10:00<4:26:35, 11.00s/it][2022-12-20 22:32:35,378] [INFO] [logging.py:68:log_dist] [Rank 0] step=3590, skipped=5, lr=[3.146666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:32:35,380] [INFO] [timer.py:197:stop] 0/7180, RunningAvgSamplesPerSec=5.877798796286612, CurrSamplesPerSec=5.346318737128904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3547/5000 [12:10:10<4:26:03, 10.99s/it][2022-12-20 22:32:46,398] [INFO] [timer.py:197:stop] 0/7182, RunningAvgSamplesPerSec=5.877798035717452, CurrSamplesPerSec=5.32761847731292, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3548/5000 [12:10:21<4:26:06, 11.00s/it][2022-12-20 22:32:57,406] [INFO] [timer.py:197:stop] 0/7184, RunningAvgSamplesPerSec=5.877798543859734, CurrSamplesPerSec=5.319566117079837, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3549/5000 [12:10:33<4:26:00, 11.00s/it][2022-12-20 22:33:08,392] [INFO] [timer.py:197:stop] 0/7186, RunningAvgSamplesPerSec=5.877802448455616, CurrSamplesPerSec=5.319569279605488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3550/5000 [12:10:43<4:25:43, 11.00s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.1400000000000004e-06, 'epoch': 86.58}

 71%|███████   | 3550/5000 [12:10:43<4:25:43, 11.00s/it][2022-12-20 22:33:19,404] [INFO] [timer.py:197:stop] 0/7188, RunningAvgSamplesPerSec=5.877802770504072, CurrSamplesPerSec=5.3441872101026995, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3551/5000 [12:10:54<4:25:39, 11.00s/it][2022-12-20 22:33:30,389] [INFO] [timer.py:197:stop] 0/7190, RunningAvgSamplesPerSec=5.877806727692985, CurrSamplesPerSec=5.361848741771878, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3552/5000 [12:11:05<4:25:21, 11.00s/it][2022-12-20 22:33:41,383] [INFO] [timer.py:197:stop] 0/7192, RunningAvgSamplesPerSec=5.8778093415532435, CurrSamplesPerSec=5.364391857433138, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3553/5000 [12:11:16<4:25:10, 11.00s/it][2022-12-20 22:33:52,385] [INFO] [timer.py:197:stop] 0/7194, RunningAvgSamplesPerSec=5.87781080382385, CurrSamplesPerSec=5.3500882448655105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3554/5000 [12:11:27<4:25:02, 11.00s/it][2022-12-20 22:34:03,380] [INFO] [timer.py:197:stop] 0/7196, RunningAvgSamplesPerSec=5.877813247252966, CurrSamplesPerSec=5.354332095802918, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3555/5000 [12:11:38<4:24:50, 11.00s/it][2022-12-20 22:34:14,406] [INFO] [timer.py:197:stop] 0/7198, RunningAvgSamplesPerSec=5.877811163977937, CurrSamplesPerSec=5.3052112586536255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3556/5000 [12:11:50<4:24:51, 11.01s/it][2022-12-20 22:34:25,413] [INFO] [logging.py:68:log_dist] [Rank 0] step=3600, skipped=5, lr=[3.124444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:34:25,414] [INFO] [timer.py:197:stop] 0/7200, RunningAvgSamplesPerSec=5.877811530666679, CurrSamplesPerSec=5.31947061057673, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3557/5000 [12:12:01<4:24:42, 11.01s/it][2022-12-20 22:34:36,461] [INFO] [timer.py:197:stop] 0/7202, RunningAvgSamplesPerSec=5.87780610863096, CurrSamplesPerSec=5.318934741446929, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3558/5000 [12:12:12<4:24:48, 11.02s/it][2022-12-20 22:34:47,410] [INFO] [timer.py:197:stop] 0/7204, RunningAvgSamplesPerSec=5.877815696622176, CurrSamplesPerSec=5.374079975527542, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3559/5000 [12:12:23<4:24:07, 11.00s/it][2022-12-20 22:34:58,442] [INFO] [timer.py:197:stop] 0/7206, RunningAvgSamplesPerSec=5.877812637694012, CurrSamplesPerSec=5.281099689782276, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3560/5000 [12:12:34<4:24:11, 11.01s/it][2022-12-20 22:35:09,454] [INFO] [timer.py:197:stop] 0/7208, RunningAvgSamplesPerSec=5.877812600273247, CurrSamplesPerSec=5.313562629067976, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3561/5000 [12:12:45<4:24:02, 11.01s/it][2022-12-20 22:35:20,437] [INFO] [timer.py:197:stop] 0/7210, RunningAvgSamplesPerSec=5.877816945700204, CurrSamplesPerSec=5.325273420837471, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████   | 3562/5000 [12:12:56<4:23:39, 11.00s/it][2022-12-20 22:35:31,435] [INFO] [timer.py:197:stop] 0/7212, RunningAvgSamplesPerSec=5.87781895269703, CurrSamplesPerSec=5.31551148109718, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3563/5000 [12:13:07<4:23:27, 11.00s/it][2022-12-20 22:35:42,476] [INFO] [timer.py:197:stop] 0/7214, RunningAvgSamplesPerSec=5.877814463094394, CurrSamplesPerSec=5.321133292918677, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3564/5000 [12:13:18<4:23:33, 11.01s/it][2022-12-20 22:35:53,527] [INFO] [timer.py:197:stop] 0/7216, RunningAvgSamplesPerSec=5.877808634928754, CurrSamplesPerSec=5.301649195523724, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3565/5000 [12:13:29<4:23:39, 11.02s/it][2022-12-20 22:36:04,536] [INFO] [timer.py:197:stop] 0/7218, RunningAvgSamplesPerSec=5.877808835922161, CurrSamplesPerSec=5.335293112179795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3566/5000 [12:13:40<4:23:22, 11.02s/it][2022-12-20 22:36:15,588] [INFO] [logging.py:68:log_dist] [Rank 0] step=3610, skipped=5, lr=[3.1022222222222225e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:36:15,589] [INFO] [timer.py:197:stop] 0/7220, RunningAvgSamplesPerSec=5.8778026151772265, CurrSamplesPerSec=5.304230470067423, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3567/5000 [12:13:51<4:23:25, 11.03s/it][2022-12-20 22:36:25,733] [INFO] [timer.py:197:stop] 0/7222, RunningAvgSamplesPerSec=5.877930934202359, CurrSamplesPerSec=5.299606281786281, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3568/5000 [12:14:06<4:52:11, 12.24s/it][2022-12-20 22:36:36,750] [INFO] [timer.py:197:stop] 0/7224, RunningAvgSamplesPerSec=5.8779300665411744, CurrSamplesPerSec=5.3113684361819296, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3569/5000 [12:14:17<4:43:30, 11.89s/it][2022-12-20 22:36:47,815] [INFO] [timer.py:197:stop] 0/7226, RunningAvgSamplesPerSec=5.8779218227636845, CurrSamplesPerSec=5.305229083092511, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3570/5000 [12:14:28<4:37:21, 11.64s/it][2022-12-20 22:36:58,812] [INFO] [timer.py:197:stop] 0/7228, RunningAvgSamplesPerSec=5.87792406941612, CurrSamplesPerSec=5.357573724468116, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3571/5000 [12:14:39<4:32:24, 11.44s/it][2022-12-20 22:37:09,821] [INFO] [timer.py:197:stop] 0/7230, RunningAvgSamplesPerSec=5.877924334691541, CurrSamplesPerSec=5.323491806475145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3572/5000 [12:14:50<4:29:03, 11.30s/it][2022-12-20 22:37:20,827] [INFO] [timer.py:197:stop] 0/7232, RunningAvgSamplesPerSec=5.877925096212432, CurrSamplesPerSec=5.3143185575724425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3573/5000 [12:15:01<4:27:07, 11.23s/it][2022-12-20 22:37:31,910] [INFO] [timer.py:197:stop] 0/7234, RunningAvgSamplesPerSec=5.877914312907908, CurrSamplesPerSec=5.292963634610542, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 71%|███████▏  | 3574/5000 [12:15:12<4:25:54, 11.19s/it][2022-12-20 22:37:42,939] [INFO] [timer.py:197:stop] 0/7236, RunningAvgSamplesPerSec=5.877911675204993, CurrSamplesPerSec=5.345352495364455, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3575/5000 [12:15:23<4:24:29, 11.14s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.084444444444445e-06, 'epoch': 87.19}

 72%|███████▏  | 3575/5000 [12:15:23<4:24:29, 11.14s/it][2022-12-20 22:37:53,979] [INFO] [timer.py:197:stop] 0/7238, RunningAvgSamplesPerSec=5.87790781182473, CurrSamplesPerSec=5.326336198063375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3576/5000 [12:15:34<4:23:32, 11.10s/it][2022-12-20 22:38:04,984] [INFO] [logging.py:68:log_dist] [Rank 0] step=3620, skipped=5, lr=[3.08e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:38:04,986] [INFO] [timer.py:197:stop] 0/7240, RunningAvgSamplesPerSec=5.877908749197606, CurrSamplesPerSec=5.345189431272094, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3577/5000 [12:15:45<4:22:39, 11.07s/it][2022-12-20 22:38:16,009] [INFO] [timer.py:197:stop] 0/7242, RunningAvgSamplesPerSec=5.877907157899447, CurrSamplesPerSec=5.328868791675421, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3578/5000 [12:15:56<4:21:55, 11.05s/it][2022-12-20 22:38:26,978] [INFO] [timer.py:197:stop] 0/7244, RunningAvgSamplesPerSec=5.877913442633043, CurrSamplesPerSec=5.354414119360696, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3579/5000 [12:16:07<4:21:23, 11.04s/it][2022-12-20 22:38:37,979] [INFO] [timer.py:197:stop] 0/7246, RunningAvgSamplesPerSec=5.877915166272913, CurrSamplesPerSec=5.355103729261289, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3580/5000 [12:16:18<4:21:02, 11.03s/it][2022-12-20 22:38:49,017] [INFO] [timer.py:197:stop] 0/7248, RunningAvgSamplesPerSec=5.877911263879821, CurrSamplesPerSec=5.333715466220601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3581/5000 [12:16:29<4:20:36, 11.02s/it][2022-12-20 22:39:00,001] [INFO] [timer.py:197:stop] 0/7250, RunningAvgSamplesPerSec=5.877915516301028, CurrSamplesPerSec=5.344528123178367, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3582/5000 [12:16:40<4:20:18, 11.01s/it][2022-12-20 22:39:11,036] [INFO] [timer.py:197:stop] 0/7252, RunningAvgSamplesPerSec=5.8779119862530385, CurrSamplesPerSec=5.316016972410149, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3583/5000 [12:16:51<4:20:17, 11.02s/it][2022-12-20 22:39:22,047] [INFO] [timer.py:197:stop] 0/7254, RunningAvgSamplesPerSec=5.877911942114626, CurrSamplesPerSec=5.338225928188709, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3584/5000 [12:17:02<4:20:10, 11.02s/it][2022-12-20 22:39:33,068] [INFO] [timer.py:197:stop] 0/7256, RunningAvgSamplesPerSec=5.8779103874626, CurrSamplesPerSec=5.347282771824609, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3585/5000 [12:17:13<4:19:30, 11.00s/it][2022-12-20 22:39:44,050] [INFO] [timer.py:197:stop] 0/7258, RunningAvgSamplesPerSec=5.877914499419193, CurrSamplesPerSec=5.324840316555253, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3586/5000 [12:17:24<4:19:11, 11.00s/it][2022-12-20 22:39:55,042] [INFO] [logging.py:68:log_dist] [Rank 0] step=3630, skipped=5, lr=[3.0577777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:39:55,043] [INFO] [timer.py:197:stop] 0/7260, RunningAvgSamplesPerSec=5.8779172340732915, CurrSamplesPerSec=5.318222173264763, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3587/5000 [12:17:35<4:19:07, 11.00s/it][2022-12-20 22:40:06,094] [INFO] [timer.py:197:stop] 0/7262, RunningAvgSamplesPerSec=5.8779112343776, CurrSamplesPerSec=5.285345063621912, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3588/5000 [12:17:46<4:19:08, 11.01s/it][2022-12-20 22:40:17,107] [INFO] [timer.py:197:stop] 0/7264, RunningAvgSamplesPerSec=5.877910911468005, CurrSamplesPerSec=5.301912026983879, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3589/5000 [12:17:57<4:19:13, 11.02s/it][2022-12-20 22:40:28,126] [INFO] [timer.py:197:stop] 0/7266, RunningAvgSamplesPerSec=5.877909984424591, CurrSamplesPerSec=5.329839241988884, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3590/5000 [12:18:08<4:18:44, 11.01s/it][2022-12-20 22:40:39,104] [INFO] [timer.py:197:stop] 0/7268, RunningAvgSamplesPerSec=5.877915049431346, CurrSamplesPerSec=5.330822326030224, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3591/5000 [12:18:19<4:18:34, 11.01s/it][2022-12-20 22:40:50,155] [INFO] [timer.py:197:stop] 0/7270, RunningAvgSamplesPerSec=5.877909400851563, CurrSamplesPerSec=5.297541314279018, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3592/5000 [12:18:30<4:18:43, 11.02s/it][2022-12-20 22:41:01,211] [INFO] [timer.py:197:stop] 0/7272, RunningAvgSamplesPerSec=5.877902804687505, CurrSamplesPerSec=5.298294573190515, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3593/5000 [12:18:41<4:18:28, 11.02s/it][2022-12-20 22:41:12,221] [INFO] [timer.py:197:stop] 0/7274, RunningAvgSamplesPerSec=5.877903397779202, CurrSamplesPerSec=5.305886364269773, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3594/5000 [12:18:52<4:18:16, 11.02s/it][2022-12-20 22:41:23,241] [INFO] [timer.py:197:stop] 0/7276, RunningAvgSamplesPerSec=5.877902025459138, CurrSamplesPerSec=5.303008241809258, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3595/5000 [12:19:03<4:18:20, 11.03s/it][2022-12-20 22:41:34,259] [INFO] [timer.py:197:stop] 0/7278, RunningAvgSamplesPerSec=5.877901068744853, CurrSamplesPerSec=5.338285802150679, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3596/5000 [12:19:14<4:17:39, 11.01s/it][2022-12-20 22:41:45,257] [INFO] [logging.py:68:log_dist] [Rank 0] step=3640, skipped=5, lr=[3.0355555555555562e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:41:45,259] [INFO] [timer.py:197:stop] 0/7280, RunningAvgSamplesPerSec=5.877902571373462, CurrSamplesPerSec=5.304908890540668, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3597/5000 [12:19:25<4:17:40, 11.02s/it][2022-12-20 22:41:56,264] [INFO] [timer.py:197:stop] 0/7282, RunningAvgSamplesPerSec=5.877903173070877, CurrSamplesPerSec=5.334335727439525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3598/5000 [12:19:36<4:17:26, 11.02s/it][2022-12-20 22:42:07,290] [INFO] [timer.py:197:stop] 0/7284, RunningAvgSamplesPerSec=5.877900954673123, CurrSamplesPerSec=5.3225249324580695, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3599/5000 [12:19:47<4:17:24, 11.02s/it][2022-12-20 22:42:18,343] [INFO] [timer.py:197:stop] 0/7286, RunningAvgSamplesPerSec=5.8778947413767355, CurrSamplesPerSec=5.312329788936401, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3600/5000 [12:19:58<4:17:07, 11.02s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 3.028888888888889e-06, 'epoch': 87.8}

 72%|███████▏  | 3600/5000 [12:19:58<4:17:07, 11.02s/it][2022-12-20 22:42:29,374] [INFO] [timer.py:197:stop] 0/7288, RunningAvgSamplesPerSec=5.877892056451469, CurrSamplesPerSec=5.29351703959998, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3601/5000 [12:20:09<4:17:12, 11.03s/it][2022-12-20 22:42:40,413] [INFO] [timer.py:197:stop] 0/7290, RunningAvgSamplesPerSec=5.877887988269642, CurrSamplesPerSec=5.308599424033712, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3602/5000 [12:20:20<4:16:54, 11.03s/it][2022-12-20 22:42:51,413] [INFO] [timer.py:197:stop] 0/7292, RunningAvgSamplesPerSec=5.877889597550178, CurrSamplesPerSec=5.320303298096859, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3603/5000 [12:20:31<4:16:53, 11.03s/it][2022-12-20 22:43:02,408] [INFO] [timer.py:197:stop] 0/7294, RunningAvgSamplesPerSec=5.877891995313751, CurrSamplesPerSec=5.3705101742920505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3604/5000 [12:20:42<4:16:05, 11.01s/it][2022-12-20 22:43:13,372] [INFO] [timer.py:197:stop] 0/7296, RunningAvgSamplesPerSec=5.877898883611844, CurrSamplesPerSec=5.353014082709496, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3605/5000 [12:20:53<4:15:45, 11.00s/it][2022-12-20 22:43:24,393] [INFO] [timer.py:197:stop] 0/7298, RunningAvgSamplesPerSec=5.877897444271325, CurrSamplesPerSec=5.321478022522421, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3606/5000 [12:21:04<4:15:58, 11.02s/it][2022-12-20 22:43:35,406] [INFO] [logging.py:68:log_dist] [Rank 0] step=3650, skipped=5, lr=[3.013333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:43:35,407] [INFO] [timer.py:197:stop] 0/7300, RunningAvgSamplesPerSec=5.877896844100633, CurrSamplesPerSec=5.360514180341661, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3607/5000 [12:21:15<4:15:35, 11.01s/it][2022-12-20 22:43:46,425] [INFO] [timer.py:197:stop] 0/7302, RunningAvgSamplesPerSec=5.87789575555824, CurrSamplesPerSec=5.334672204241832, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3608/5000 [12:21:26<4:15:21, 11.01s/it][2022-12-20 22:43:56,543] [INFO] [timer.py:197:stop] 0/7304, RunningAvgSamplesPerSec=5.878026146968897, CurrSamplesPerSec=6.2443486339058145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 22:44:07,539] [INFO] [timer.py:197:stop] 0/7306, RunningAvgSamplesPerSec=5.878027842535741, CurrSamplesPerSec=5.334809393795689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3609/5000 [12:21:43<4:51:06, 12.56s/it][2022-12-20 22:44:18,522] [INFO] [timer.py:197:stop] 0/7308, RunningAvgSamplesPerSec=5.878031980968481, CurrSamplesPerSec=5.316654394818856, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3610/5000 [12:21:54<4:39:57, 12.08s/it][2022-12-20 22:44:29,506] [INFO] [timer.py:197:stop] 0/7310, RunningAvgSamplesPerSec=5.878036092730679, CurrSamplesPerSec=5.364719056554497, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3611/5000 [12:22:05<4:32:06, 11.75s/it][2022-12-20 22:44:40,529] [INFO] [timer.py:197:stop] 0/7312, RunningAvgSamplesPerSec=5.878034498206484, CurrSamplesPerSec=5.331615365783328, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3612/5000 [12:22:16<4:26:50, 11.53s/it][2022-12-20 22:44:51,566] [INFO] [timer.py:197:stop] 0/7314, RunningAvgSamplesPerSec=5.878030823276953, CurrSamplesPerSec=5.3287898761283525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3613/5000 [12:22:27<4:23:11, 11.39s/it][2022-12-20 22:45:02,458] [INFO] [timer.py:197:stop] 0/7316, RunningAvgSamplesPerSec=5.87804834884731, CurrSamplesPerSec=5.40590934456368, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3614/5000 [12:22:38<4:19:35, 11.24s/it][2022-12-20 22:45:13,439] [INFO] [timer.py:197:stop] 0/7318, RunningAvgSamplesPerSec=5.878052890057602, CurrSamplesPerSec=5.371450063538423, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3615/5000 [12:22:49<4:17:37, 11.16s/it][2022-12-20 22:45:24,444] [INFO] [logging.py:68:log_dist] [Rank 0] step=3660, skipped=5, lr=[2.9911111111111115e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:45:24,446] [INFO] [timer.py:197:stop] 0/7320, RunningAvgSamplesPerSec=5.8780533728260975, CurrSamplesPerSec=5.337269400867251, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3616/5000 [12:23:00<4:16:22, 11.11s/it][2022-12-20 22:45:35,463] [INFO] [timer.py:197:stop] 0/7322, RunningAvgSamplesPerSec=5.878052364916596, CurrSamplesPerSec=5.331462245204065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3617/5000 [12:23:11<4:15:31, 11.09s/it][2022-12-20 22:45:46,453] [INFO] [timer.py:197:stop] 0/7324, RunningAvgSamplesPerSec=5.878055378081517, CurrSamplesPerSec=5.331975646815904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3618/5000 [12:23:22<4:14:40, 11.06s/it][2022-12-20 22:45:57,441] [INFO] [timer.py:197:stop] 0/7326, RunningAvgSamplesPerSec=5.8780586345905075, CurrSamplesPerSec=5.3712213476487225, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3619/5000 [12:23:33<4:14:00, 11.04s/it][2022-12-20 22:46:08,466] [INFO] [timer.py:197:stop] 0/7328, RunningAvgSamplesPerSec=5.878056452421843, CurrSamplesPerSec=5.330106992728701, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3620/5000 [12:23:44<4:13:45, 11.03s/it][2022-12-20 22:46:19,511] [INFO] [timer.py:197:stop] 0/7330, RunningAvgSamplesPerSec=5.87805140526741, CurrSamplesPerSec=5.318821973930595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3621/5000 [12:23:55<4:13:39, 11.04s/it][2022-12-20 22:46:30,503] [INFO] [timer.py:197:stop] 0/7332, RunningAvgSamplesPerSec=5.8780541307577785, CurrSamplesPerSec=5.3348390802537216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3622/5000 [12:24:06<4:13:09, 11.02s/it][2022-12-20 22:46:41,472] [INFO] [timer.py:197:stop] 0/7334, RunningAvgSamplesPerSec=5.878060196639519, CurrSamplesPerSec=5.350546156018779, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3623/5000 [12:24:17<4:12:36, 11.01s/it][2022-12-20 22:46:52,493] [INFO] [timer.py:197:stop] 0/7336, RunningAvgSamplesPerSec=5.878058521840436, CurrSamplesPerSec=5.340344202190716, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▏  | 3624/5000 [12:24:28<4:12:31, 11.01s/it][2022-12-20 22:47:03,563] [INFO] [timer.py:197:stop] 0/7338, RunningAvgSamplesPerSec=5.878049806912412, CurrSamplesPerSec=5.310230104766655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 72%|███████▎  | 3625/5000 [12:24:39<4:12:44, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.9711111111111112e-06, 'epoch': 88.41}

 72%|███████▎  | 3625/5000 [12:24:39<4:12:44, 11.03s/it][2022-12-20 22:47:14,558] [INFO] [logging.py:68:log_dist] [Rank 0] step=3670, skipped=5, lr=[2.968888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:47:14,560] [INFO] [timer.py:197:stop] 0/7340, RunningAvgSamplesPerSec=5.878052333548965, CurrSamplesPerSec=5.347223974072178, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3626/5000 [12:24:50<4:12:20, 11.02s/it][2022-12-20 22:47:25,601] [INFO] [timer.py:197:stop] 0/7342, RunningAvgSamplesPerSec=5.878047900381191, CurrSamplesPerSec=5.303931776894248, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3627/5000 [12:25:01<4:12:18, 11.03s/it][2022-12-20 22:47:36,614] [INFO] [timer.py:197:stop] 0/7344, RunningAvgSamplesPerSec=5.878047730698672, CurrSamplesPerSec=5.331107750811973, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3628/5000 [12:25:12<4:12:01, 11.02s/it][2022-12-20 22:47:47,738] [INFO] [timer.py:197:stop] 0/7346, RunningAvgSamplesPerSec=5.8780310561933975, CurrSamplesPerSec=5.235020312277251, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3629/5000 [12:25:23<4:12:33, 11.05s/it][2022-12-20 22:47:58,765] [INFO] [timer.py:197:stop] 0/7348, RunningAvgSamplesPerSec=5.878028691253228, CurrSamplesPerSec=5.327920901860552, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3630/5000 [12:25:34<4:12:11, 11.05s/it][2022-12-20 22:48:09,788] [INFO] [timer.py:197:stop] 0/7350, RunningAvgSamplesPerSec=5.87802699583217, CurrSamplesPerSec=5.333532764686408, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3631/5000 [12:25:45<4:11:51, 11.04s/it][2022-12-20 22:48:20,777] [INFO] [timer.py:197:stop] 0/7352, RunningAvgSamplesPerSec=5.878030309104839, CurrSamplesPerSec=5.3392248525194175, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3632/5000 [12:25:56<4:11:19, 11.02s/it][2022-12-20 22:48:31,735] [INFO] [timer.py:197:stop] 0/7354, RunningAvgSamplesPerSec=5.878038024013735, CurrSamplesPerSec=5.363030957343603, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3633/5000 [12:26:07<4:10:42, 11.00s/it][2022-12-20 22:48:42,797] [INFO] [timer.py:197:stop] 0/7356, RunningAvgSamplesPerSec=5.878030863530207, CurrSamplesPerSec=5.301377176699093, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3634/5000 [12:26:18<4:10:54, 11.02s/it][2022-12-20 22:48:53,867] [INFO] [timer.py:197:stop] 0/7358, RunningAvgSamplesPerSec=5.87802229856793, CurrSamplesPerSec=5.300160658162993, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3635/5000 [12:26:29<4:11:03, 11.04s/it][2022-12-20 22:49:04,926] [INFO] [logging.py:68:log_dist] [Rank 0] step=3680, skipped=5, lr=[2.946666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:49:04,928] [INFO] [timer.py:197:stop] 0/7360, RunningAvgSamplesPerSec=5.878015273498023, CurrSamplesPerSec=5.301714743879037, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3636/5000 [12:26:40<4:11:03, 11.04s/it][2022-12-20 22:49:15,949] [INFO] [timer.py:197:stop] 0/7362, RunningAvgSamplesPerSec=5.878014029393253, CurrSamplesPerSec=5.326591547569705, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3637/5000 [12:26:51<4:10:43, 11.04s/it][2022-12-20 22:49:26,968] [INFO] [timer.py:197:stop] 0/7364, RunningAvgSamplesPerSec=5.878012949924097, CurrSamplesPerSec=5.322669730089044, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3638/5000 [12:27:02<4:10:24, 11.03s/it][2022-12-20 22:49:37,947] [INFO] [timer.py:197:stop] 0/7366, RunningAvgSamplesPerSec=5.878017645845158, CurrSamplesPerSec=5.3439699592295415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3639/5000 [12:27:13<4:09:52, 11.02s/it][2022-12-20 22:49:48,934] [INFO] [timer.py:197:stop] 0/7368, RunningAvgSamplesPerSec=5.8780212121197195, CurrSamplesPerSec=5.313443358020031, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3640/5000 [12:27:24<4:09:29, 11.01s/it][2022-12-20 22:50:00,024] [INFO] [timer.py:197:stop] 0/7370, RunningAvgSamplesPerSec=5.878009794764269, CurrSamplesPerSec=5.283415145251304, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3641/5000 [12:27:35<4:09:52, 11.03s/it][2022-12-20 22:50:11,028] [INFO] [timer.py:197:stop] 0/7372, RunningAvgSamplesPerSec=5.878010841181725, CurrSamplesPerSec=5.33514444585645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3642/5000 [12:27:46<4:09:30, 11.02s/it][2022-12-20 22:50:21,968] [INFO] [timer.py:197:stop] 0/7374, RunningAvgSamplesPerSec=5.8780214076935575, CurrSamplesPerSec=5.358752985652896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3643/5000 [12:27:57<4:08:44, 11.00s/it][2022-12-20 22:50:32,917] [INFO] [timer.py:197:stop] 0/7376, RunningAvgSamplesPerSec=5.8780305800332115, CurrSamplesPerSec=5.36434790515311, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3644/5000 [12:28:08<4:08:13, 10.98s/it][2022-12-20 22:50:43,847] [INFO] [timer.py:197:stop] 0/7378, RunningAvgSamplesPerSec=5.878042347007752, CurrSamplesPerSec=5.369321222619378, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3645/5000 [12:28:19<4:07:41, 10.97s/it][2022-12-20 22:50:54,832] [INFO] [logging.py:68:log_dist] [Rank 0] step=3690, skipped=5, lr=[2.9244444444444447e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:50:54,833] [INFO] [timer.py:197:stop] 0/7380, RunningAvgSamplesPerSec=5.878045830509898, CurrSamplesPerSec=5.331595033913721, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3646/5000 [12:28:30<4:07:37, 10.97s/it][2022-12-20 22:51:05,792] [INFO] [timer.py:197:stop] 0/7382, RunningAvgSamplesPerSec=5.87805330293477, CurrSamplesPerSec=5.371309263388221, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3647/5000 [12:28:41<4:07:20, 10.97s/it][2022-12-20 22:51:16,824] [INFO] [timer.py:197:stop] 0/7384, RunningAvgSamplesPerSec=5.878050257471396, CurrSamplesPerSec=5.304262332658203, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3648/5000 [12:28:52<4:07:35, 10.99s/it][2022-12-20 22:51:27,805] [INFO] [timer.py:197:stop] 0/7386, RunningAvgSamplesPerSec=5.878054638963181, CurrSamplesPerSec=5.353333062644151, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3649/5000 [12:29:03<4:07:21, 10.99s/it][2022-12-20 22:51:37,930] [INFO] [timer.py:197:stop] 0/7388, RunningAvgSamplesPerSec=5.878182748346063, CurrSamplesPerSec=5.3158426396809615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3650/5000 [12:29:18<4:34:26, 12.20s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.915555555555556e-06, 'epoch': 89.02}

 73%|███████▎  | 3650/5000 [12:29:18<4:34:26, 12.20s/it][2022-12-20 22:51:48,897] [INFO] [timer.py:197:stop] 0/7390, RunningAvgSamplesPerSec=5.87818926627693, CurrSamplesPerSec=5.332236833349364, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3651/5000 [12:29:29<4:26:09, 11.84s/it][2022-12-20 22:51:59,920] [INFO] [timer.py:197:stop] 0/7392, RunningAvgSamplesPerSec=5.878187285979901, CurrSamplesPerSec=5.309254180356859, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3652/5000 [12:29:40<4:20:22, 11.59s/it][2022-12-20 22:52:10,883] [INFO] [timer.py:197:stop] 0/7394, RunningAvgSamplesPerSec=5.8781941906563215, CurrSamplesPerSec=5.350409862205678, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3653/5000 [12:29:51<4:15:50, 11.40s/it][2022-12-20 22:52:21,846] [INFO] [timer.py:197:stop] 0/7396, RunningAvgSamplesPerSec=5.878200977133208, CurrSamplesPerSec=5.335010844103868, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3654/5000 [12:30:02<4:12:59, 11.28s/it][2022-12-20 22:52:32,817] [INFO] [timer.py:197:stop] 0/7398, RunningAvgSamplesPerSec=5.878206764855284, CurrSamplesPerSec=5.361465137223794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3655/5000 [12:30:13<4:10:29, 11.17s/it][2022-12-20 22:52:43,798] [INFO] [logging.py:68:log_dist] [Rank 0] step=3700, skipped=5, lr=[2.9022222222222223e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:52:43,800] [INFO] [timer.py:197:stop] 0/7400, RunningAvgSamplesPerSec=5.878210684548515, CurrSamplesPerSec=5.317834461275866, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3656/5000 [12:30:24<4:09:16, 11.13s/it][2022-12-20 22:52:54,847] [INFO] [timer.py:197:stop] 0/7402, RunningAvgSamplesPerSec=5.8782050604732845, CurrSamplesPerSec=5.292844451623324, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3657/5000 [12:30:35<4:08:37, 11.11s/it][2022-12-20 22:53:05,824] [INFO] [timer.py:197:stop] 0/7404, RunningAvgSamplesPerSec=5.8782097900660215, CurrSamplesPerSec=5.366779869972342, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3658/5000 [12:30:46<4:07:29, 11.07s/it][2022-12-20 22:53:16,829] [INFO] [timer.py:197:stop] 0/7406, RunningAvgSamplesPerSec=5.878210688049898, CurrSamplesPerSec=5.3328508177871665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3659/5000 [12:30:57<4:06:55, 11.05s/it][2022-12-20 22:53:27,842] [INFO] [timer.py:197:stop] 0/7408, RunningAvgSamplesPerSec=5.878210425180106, CurrSamplesPerSec=5.328186979257869, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3660/5000 [12:31:08<4:06:40, 11.04s/it][2022-12-20 22:53:38,858] [INFO] [timer.py:197:stop] 0/7410, RunningAvgSamplesPerSec=5.878209737227267, CurrSamplesPerSec=5.347450437733798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3661/5000 [12:31:19<4:06:14, 11.03s/it][2022-12-20 22:53:49,874] [INFO] [timer.py:197:stop] 0/7412, RunningAvgSamplesPerSec=5.878208894589859, CurrSamplesPerSec=5.34074455425436, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3662/5000 [12:31:30<4:05:54, 11.03s/it][2022-12-20 22:54:00,846] [INFO] [timer.py:197:stop] 0/7414, RunningAvgSamplesPerSec=5.878214784505318, CurrSamplesPerSec=5.376476838677448, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3663/5000 [12:31:41<4:05:07, 11.00s/it][2022-12-20 22:54:11,855] [INFO] [timer.py:197:stop] 0/7416, RunningAvgSamplesPerSec=5.87821516827551, CurrSamplesPerSec=5.313422323117286, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3664/5000 [12:31:52<4:05:08, 11.01s/it][2022-12-20 22:54:22,887] [INFO] [timer.py:197:stop] 0/7418, RunningAvgSamplesPerSec=5.878212142178648, CurrSamplesPerSec=5.312883886306952, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3665/5000 [12:32:03<4:05:06, 11.02s/it][2022-12-20 22:54:33,877] [INFO] [logging.py:68:log_dist] [Rank 0] step=3710, skipped=5, lr=[2.88e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:54:33,878] [INFO] [timer.py:197:stop] 0/7420, RunningAvgSamplesPerSec=5.8782150698215725, CurrSamplesPerSec=5.348794275202794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3666/5000 [12:32:14<4:04:31, 11.00s/it][2022-12-20 22:54:44,885] [INFO] [timer.py:197:stop] 0/7422, RunningAvgSamplesPerSec=5.878215546432252, CurrSamplesPerSec=5.301923546090147, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3667/5000 [12:32:25<4:04:40, 11.01s/it][2022-12-20 22:54:55,918] [INFO] [timer.py:197:stop] 0/7424, RunningAvgSamplesPerSec=5.878212275573226, CurrSamplesPerSec=5.314704704789132, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3668/5000 [12:32:36<4:04:36, 11.02s/it][2022-12-20 22:55:06,958] [INFO] [timer.py:197:stop] 0/7426, RunningAvgSamplesPerSec=5.878208159805311, CurrSamplesPerSec=5.30888373391323, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3669/5000 [12:32:47<4:04:33, 11.02s/it][2022-12-20 22:55:17,944] [INFO] [timer.py:197:stop] 0/7428, RunningAvgSamplesPerSec=5.878211773794939, CurrSamplesPerSec=5.355146248200109, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3670/5000 [12:32:58<4:03:59, 11.01s/it][2022-12-20 22:55:28,976] [INFO] [timer.py:197:stop] 0/7430, RunningAvgSamplesPerSec=5.878208823479481, CurrSamplesPerSec=5.297335575075926, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3671/5000 [12:33:09<4:04:08, 11.02s/it][2022-12-20 22:55:40,014] [INFO] [timer.py:197:stop] 0/7432, RunningAvgSamplesPerSec=5.878204866157317, CurrSamplesPerSec=5.3134643930893235, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3672/5000 [12:33:20<4:04:01, 11.02s/it][2022-12-20 22:55:51,033] [INFO] [timer.py:197:stop] 0/7434, RunningAvgSamplesPerSec=5.878203642722273, CurrSamplesPerSec=5.324958198449896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3673/5000 [12:33:31<4:03:57, 11.03s/it][2022-12-20 22:56:02,090] [INFO] [timer.py:197:stop] 0/7436, RunningAvgSamplesPerSec=5.878196980080248, CurrSamplesPerSec=5.313976227650446, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 73%|███████▎  | 3674/5000 [12:33:42<4:03:57, 11.04s/it][2022-12-20 22:56:13,161] [INFO] [timer.py:197:stop] 0/7438, RunningAvgSamplesPerSec=5.878188150274712, CurrSamplesPerSec=5.300909637589686, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3675/5000 [12:33:53<4:03:35, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.86e-06, 'epoch': 89.63}

 74%|███████▎  | 3675/5000 [12:33:53<4:03:35, 11.03s/it][2022-12-20 22:56:24,182] [INFO] [logging.py:68:log_dist] [Rank 0] step=3720, skipped=5, lr=[2.8577777777777784e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:56:24,183] [INFO] [timer.py:197:stop] 0/7440, RunningAvgSamplesPerSec=5.878186611237144, CurrSamplesPerSec=5.292870542011028, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3676/5000 [12:34:04<4:03:34, 11.04s/it][2022-12-20 22:56:35,228] [INFO] [timer.py:197:stop] 0/7442, RunningAvgSamplesPerSec=5.878181684090456, CurrSamplesPerSec=5.302484901371005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3677/5000 [12:34:15<4:03:34, 11.05s/it][2022-12-20 22:56:46,248] [INFO] [timer.py:197:stop] 0/7444, RunningAvgSamplesPerSec=5.878180223552527, CurrSamplesPerSec=5.342893115329626, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3678/5000 [12:34:26<4:03:06, 11.03s/it][2022-12-20 22:56:57,259] [INFO] [timer.py:197:stop] 0/7446, RunningAvgSamplesPerSec=5.87818018684361, CurrSamplesPerSec=5.335865798835011, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3679/5000 [12:34:37<4:02:53, 11.03s/it][2022-12-20 22:57:08,279] [INFO] [timer.py:197:stop] 0/7448, RunningAvgSamplesPerSec=5.878178891789989, CurrSamplesPerSec=5.3446281496239285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3680/5000 [12:34:48<4:02:38, 11.03s/it][2022-12-20 22:57:19,324] [INFO] [timer.py:197:stop] 0/7450, RunningAvgSamplesPerSec=5.878173834211167, CurrSamplesPerSec=5.321941389215166, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3681/5000 [12:34:59<4:02:17, 11.02s/it][2022-12-20 22:57:30,341] [INFO] [timer.py:197:stop] 0/7452, RunningAvgSamplesPerSec=5.878172652450014, CurrSamplesPerSec=5.311729769820751, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3682/5000 [12:35:10<4:01:58, 11.02s/it][2022-12-20 22:57:41,355] [INFO] [timer.py:197:stop] 0/7454, RunningAvgSamplesPerSec=5.878172186743465, CurrSamplesPerSec=5.300643556096169, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3683/5000 [12:35:21<4:02:01, 11.03s/it][2022-12-20 22:57:52,347] [INFO] [timer.py:197:stop] 0/7456, RunningAvgSamplesPerSec=5.878174806794226, CurrSamplesPerSec=5.353260253300206, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3684/5000 [12:35:32<4:01:42, 11.02s/it][2022-12-20 22:58:03,344] [INFO] [timer.py:197:stop] 0/7458, RunningAvgSamplesPerSec=5.8781769894239915, CurrSamplesPerSec=5.363932002158878, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3685/5000 [12:35:43<4:01:16, 11.01s/it][2022-12-20 22:58:14,362] [INFO] [logging.py:68:log_dist] [Rank 0] step=3730, skipped=5, lr=[2.835555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 22:58:14,363] [INFO] [timer.py:197:stop] 0/7460, RunningAvgSamplesPerSec=5.878175673460642, CurrSamplesPerSec=5.328023691727129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3686/5000 [12:35:54<4:00:55, 11.00s/it][2022-12-20 22:58:25,345] [INFO] [timer.py:197:stop] 0/7462, RunningAvgSamplesPerSec=5.8781799166570705, CurrSamplesPerSec=5.331557971008736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▎  | 3687/5000 [12:36:05<4:01:00, 11.01s/it][2022-12-20 22:58:36,400] [INFO] [timer.py:197:stop] 0/7464, RunningAvgSamplesPerSec=5.8781736174222, CurrSamplesPerSec=5.320273562299765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3688/5000 [12:36:16<4:00:48, 11.01s/it][2022-12-20 22:58:47,412] [INFO] [timer.py:197:stop] 0/7466, RunningAvgSamplesPerSec=5.878173538778539, CurrSamplesPerSec=5.3193152353601025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3689/5000 [12:36:27<4:00:22, 11.00s/it][2022-12-20 22:58:58,391] [INFO] [timer.py:197:stop] 0/7468, RunningAvgSamplesPerSec=5.878178142466483, CurrSamplesPerSec=5.3138081297177076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3690/5000 [12:36:38<4:00:19, 11.01s/it][2022-12-20 22:59:08,491] [INFO] [timer.py:197:stop] 0/7470, RunningAvgSamplesPerSec=5.878308404306301, CurrSamplesPerSec=6.261764181850644, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 22:59:19,523] [INFO] [timer.py:197:stop] 0/7472, RunningAvgSamplesPerSec=5.878304989022367, CurrSamplesPerSec=5.31860825571978, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3691/5000 [12:36:55<4:34:06, 12.56s/it][2022-12-20 22:59:30,546] [INFO] [timer.py:197:stop] 0/7474, RunningAvgSamplesPerSec=5.878303353055128, CurrSamplesPerSec=5.311923594391239, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3692/5000 [12:37:06<4:23:48, 12.10s/it][2022-12-20 22:59:41,509] [INFO] [timer.py:197:stop] 0/7476, RunningAvgSamplesPerSec=5.878310437952915, CurrSamplesPerSec=5.353786190523981, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3693/5000 [12:37:17<4:16:10, 11.76s/it][2022-12-20 22:59:52,580] [INFO] [timer.py:197:stop] 0/7478, RunningAvgSamplesPerSec=5.878301738271722, CurrSamplesPerSec=5.305817776268907, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3694/5000 [12:37:28<4:11:28, 11.55s/it][2022-12-20 23:00:03,589] [INFO] [logging.py:68:log_dist] [Rank 0] step=3740, skipped=5, lr=[2.8133333333333336e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:00:03,590] [INFO] [timer.py:197:stop] 0/7480, RunningAvgSamplesPerSec=5.878301723692942, CurrSamplesPerSec=5.3055344238442705, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3695/5000 [12:37:39<4:07:44, 11.39s/it][2022-12-20 23:00:14,618] [INFO] [timer.py:197:stop] 0/7482, RunningAvgSamplesPerSec=5.878299272050411, CurrSamplesPerSec=5.329698075157004, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3696/5000 [12:37:50<4:05:11, 11.28s/it][2022-12-20 23:00:25,593] [INFO] [timer.py:197:stop] 0/7484, RunningAvgSamplesPerSec=5.878304467366728, CurrSamplesPerSec=5.370003935348891, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3697/5000 [12:38:01<4:03:00, 11.19s/it][2022-12-20 23:00:36,624] [INFO] [timer.py:197:stop] 0/7486, RunningAvgSamplesPerSec=5.878301585058994, CurrSamplesPerSec=5.30574394651884, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3698/5000 [12:38:12<4:01:46, 11.14s/it][2022-12-20 23:00:47,708] [INFO] [timer.py:197:stop] 0/7488, RunningAvgSamplesPerSec=5.878291097964548, CurrSamplesPerSec=5.283247519455194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3699/5000 [12:38:23<4:01:13, 11.12s/it][2022-12-20 23:00:58,793] [INFO] [timer.py:197:stop] 0/7490, RunningAvgSamplesPerSec=5.87828052563925, CurrSamplesPerSec=5.313727135136402, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3700/5000 [12:38:34<4:00:46, 11.11s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.8022222222222225e-06, 'epoch': 90.24}

 74%|███████▍  | 3700/5000 [12:38:34<4:00:46, 11.11s/it][2022-12-20 23:01:09,877] [INFO] [timer.py:197:stop] 0/7492, RunningAvgSamplesPerSec=5.878270118897539, CurrSamplesPerSec=5.299302459905826, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3701/5000 [12:38:45<4:00:24, 11.10s/it][2022-12-20 23:01:20,906] [INFO] [timer.py:197:stop] 0/7494, RunningAvgSamplesPerSec=5.878267525492624, CurrSamplesPerSec=5.3045229075522045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3702/5000 [12:38:56<3:59:43, 11.08s/it][2022-12-20 23:01:31,958] [INFO] [timer.py:197:stop] 0/7496, RunningAvgSamplesPerSec=5.878261753173252, CurrSamplesPerSec=5.300910265665725, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3703/5000 [12:39:07<3:59:21, 11.07s/it][2022-12-20 23:01:42,897] [INFO] [timer.py:197:stop] 0/7498, RunningAvgSamplesPerSec=5.878272261372583, CurrSamplesPerSec=5.352530347669919, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3704/5000 [12:39:18<3:58:18, 11.03s/it][2022-12-20 23:01:53,908] [INFO] [logging.py:68:log_dist] [Rank 0] step=3750, skipped=5, lr=[2.7911111111111113e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:01:53,910] [INFO] [timer.py:197:stop] 0/7500, RunningAvgSamplesPerSec=5.878271941932843, CurrSamplesPerSec=5.351049586155769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3705/5000 [12:39:29<3:57:59, 11.03s/it][2022-12-20 23:02:04,874] [INFO] [timer.py:197:stop] 0/7502, RunningAvgSamplesPerSec=5.878278677706603, CurrSamplesPerSec=5.341788641959979, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3706/5000 [12:39:40<3:57:24, 11.01s/it][2022-12-20 23:02:15,874] [INFO] [timer.py:197:stop] 0/7504, RunningAvgSamplesPerSec=5.8782803357034235, CurrSamplesPerSec=5.358951541034085, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3707/5000 [12:39:51<3:57:10, 11.01s/it][2022-12-20 23:02:26,928] [INFO] [timer.py:197:stop] 0/7506, RunningAvgSamplesPerSec=5.878274027798656, CurrSamplesPerSec=5.303792188672462, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3708/5000 [12:40:02<3:57:18, 11.02s/it][2022-12-20 23:02:37,968] [INFO] [timer.py:197:stop] 0/7508, RunningAvgSamplesPerSec=5.878269937813006, CurrSamplesPerSec=5.3118720886514295, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3709/5000 [12:40:13<3:57:14, 11.03s/it][2022-12-20 23:02:49,016] [INFO] [timer.py:197:stop] 0/7510, RunningAvgSamplesPerSec=5.878265098243694, CurrSamplesPerSec=5.333810637058999, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3710/5000 [12:40:24<3:57:12, 11.03s/it][2022-12-20 23:03:00,003] [INFO] [timer.py:197:stop] 0/7512, RunningAvgSamplesPerSec=5.878268643141073, CurrSamplesPerSec=5.337335832986836, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3711/5000 [12:40:35<3:56:43, 11.02s/it][2022-12-20 23:03:11,036] [INFO] [timer.py:197:stop] 0/7514, RunningAvgSamplesPerSec=5.878265530385083, CurrSamplesPerSec=5.315779689005386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3712/5000 [12:40:46<3:56:38, 11.02s/it][2022-12-20 23:03:22,004] [INFO] [timer.py:197:stop] 0/7516, RunningAvgSamplesPerSec=5.878271810608499, CurrSamplesPerSec=5.3113722195255555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3713/5000 [12:40:57<3:56:05, 11.01s/it][2022-12-20 23:03:33,052] [INFO] [timer.py:197:stop] 0/7518, RunningAvgSamplesPerSec=5.87826670738317, CurrSamplesPerSec=5.318396662104756, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3714/5000 [12:41:08<3:56:10, 11.02s/it][2022-12-20 23:03:44,036] [INFO] [logging.py:68:log_dist] [Rank 0] step=3760, skipped=5, lr=[2.7688888888888893e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:03:44,037] [INFO] [timer.py:197:stop] 0/7520, RunningAvgSamplesPerSec=5.878270256643285, CurrSamplesPerSec=5.318512783607798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3715/5000 [12:41:19<3:55:46, 11.01s/it][2022-12-20 23:03:55,081] [INFO] [timer.py:197:stop] 0/7522, RunningAvgSamplesPerSec=5.8782656238565165, CurrSamplesPerSec=5.320530439939047, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3716/5000 [12:41:30<3:55:48, 11.02s/it][2022-12-20 23:04:06,100] [INFO] [timer.py:197:stop] 0/7524, RunningAvgSamplesPerSec=5.878264496594915, CurrSamplesPerSec=5.330162239509823, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3717/5000 [12:41:41<3:55:37, 11.02s/it][2022-12-20 23:04:17,091] [INFO] [timer.py:197:stop] 0/7526, RunningAvgSamplesPerSec=5.878267221105005, CurrSamplesPerSec=5.330617381093487, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3718/5000 [12:41:52<3:55:16, 11.01s/it][2022-12-20 23:04:28,102] [INFO] [timer.py:197:stop] 0/7528, RunningAvgSamplesPerSec=5.8782673045270135, CurrSamplesPerSec=5.344393625434488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3719/5000 [12:42:03<3:55:04, 11.01s/it][2022-12-20 23:04:39,135] [INFO] [timer.py:197:stop] 0/7530, RunningAvgSamplesPerSec=5.878264161433213, CurrSamplesPerSec=5.313576092097527, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3720/5000 [12:42:14<3:55:02, 11.02s/it][2022-12-20 23:04:50,091] [INFO] [timer.py:197:stop] 0/7532, RunningAvgSamplesPerSec=5.8782720957249825, CurrSamplesPerSec=5.361969767438701, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3721/5000 [12:42:25<3:54:27, 11.00s/it][2022-12-20 23:05:01,073] [INFO] [timer.py:197:stop] 0/7534, RunningAvgSamplesPerSec=5.878276078235258, CurrSamplesPerSec=5.337995362284487, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3722/5000 [12:42:36<3:54:10, 10.99s/it][2022-12-20 23:05:12,008] [INFO] [timer.py:197:stop] 0/7536, RunningAvgSamplesPerSec=5.878286766695985, CurrSamplesPerSec=5.362018179235223, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3723/5000 [12:42:47<3:53:36, 10.98s/it][2022-12-20 23:05:22,961] [INFO] [timer.py:197:stop] 0/7538, RunningAvgSamplesPerSec=5.878295039241417, CurrSamplesPerSec=5.361974480053053, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3724/5000 [12:42:58<3:53:16, 10.97s/it][2022-12-20 23:05:34,006] [INFO] [logging.py:68:log_dist] [Rank 0] step=3770, skipped=5, lr=[2.746666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:05:34,007] [INFO] [timer.py:197:stop] 0/7540, RunningAvgSamplesPerSec=5.87828989330566, CurrSamplesPerSec=5.278788339068505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 74%|███████▍  | 3725/5000 [12:43:09<3:53:35, 10.99s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.746666666666667e-06, 'epoch': 90.84}

 74%|███████▍  | 3725/5000 [12:43:09<3:53:35, 10.99s/it][2022-12-20 23:05:44,968] [INFO] [timer.py:197:stop] 0/7542, RunningAvgSamplesPerSec=5.878297361248722, CurrSamplesPerSec=5.361328286574438, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3726/5000 [12:43:20<3:53:12, 10.98s/it][2022-12-20 23:05:55,908] [INFO] [timer.py:197:stop] 0/7544, RunningAvgSamplesPerSec=5.878307531489085, CurrSamplesPerSec=5.375669752221967, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3727/5000 [12:43:31<3:52:44, 10.97s/it][2022-12-20 23:06:06,902] [INFO] [timer.py:197:stop] 0/7546, RunningAvgSamplesPerSec=5.8783096388257565, CurrSamplesPerSec=5.3144612255936305, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3728/5000 [12:43:42<3:52:43, 10.98s/it][2022-12-20 23:06:17,926] [INFO] [timer.py:197:stop] 0/7548, RunningAvgSamplesPerSec=5.878307625348882, CurrSamplesPerSec=5.337223557301823, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3729/5000 [12:43:53<3:52:49, 10.99s/it][2022-12-20 23:06:28,937] [INFO] [timer.py:197:stop] 0/7550, RunningAvgSamplesPerSec=5.878307688067831, CurrSamplesPerSec=5.339900994920523, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3730/5000 [12:44:04<3:52:46, 11.00s/it][2022-12-20 23:06:39,903] [INFO] [timer.py:197:stop] 0/7552, RunningAvgSamplesPerSec=5.8783138604667, CurrSamplesPerSec=5.355189408818497, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3731/5000 [12:44:15<3:52:23, 10.99s/it][2022-12-20 23:06:49,969] [INFO] [timer.py:197:stop] 0/7554, RunningAvgSamplesPerSec=5.878447486439268, CurrSamplesPerSec=5.359047186646252, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3732/5000 [12:44:30<4:17:31, 12.19s/it][2022-12-20 23:07:00,979] [INFO] [timer.py:197:stop] 0/7556, RunningAvgSamplesPerSec=5.878447592823564, CurrSamplesPerSec=5.30527689512011, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3733/5000 [12:44:41<4:09:49, 11.83s/it][2022-12-20 23:07:11,965] [INFO] [timer.py:197:stop] 0/7558, RunningAvgSamplesPerSec=5.8784509578912365, CurrSamplesPerSec=5.320025566045121, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3734/5000 [12:44:52<4:04:37, 11.59s/it][2022-12-20 23:07:23,013] [INFO] [logging.py:68:log_dist] [Rank 0] step=3780, skipped=5, lr=[2.7244444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:07:23,014] [INFO] [timer.py:197:stop] 0/7560, RunningAvgSamplesPerSec=5.878445182145888, CurrSamplesPerSec=5.312091573111797, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3735/5000 [12:45:03<4:00:41, 11.42s/it][2022-12-20 23:07:34,031] [INFO] [timer.py:197:stop] 0/7562, RunningAvgSamplesPerSec=5.878444256546486, CurrSamplesPerSec=5.2989787981987435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3736/5000 [12:45:14<3:58:10, 11.31s/it][2022-12-20 23:07:45,076] [INFO] [timer.py:197:stop] 0/7564, RunningAvgSamplesPerSec=5.878439404771691, CurrSamplesPerSec=5.303660781250123, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3737/5000 [12:45:25<3:56:06, 11.22s/it][2022-12-20 23:07:56,041] [INFO] [timer.py:197:stop] 0/7566, RunningAvgSamplesPerSec=5.87844571088064, CurrSamplesPerSec=5.340764743448151, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3738/5000 [12:45:36<3:54:36, 11.15s/it][2022-12-20 23:08:07,067] [INFO] [timer.py:197:stop] 0/7568, RunningAvgSamplesPerSec=5.8784435764863625, CurrSamplesPerSec=5.324779053832397, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3739/5000 [12:45:47<3:53:02, 11.09s/it][2022-12-20 23:08:18,021] [INFO] [timer.py:197:stop] 0/7570, RunningAvgSamplesPerSec=5.878451607713328, CurrSamplesPerSec=5.308075609284219, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3740/5000 [12:45:58<3:52:46, 11.08s/it][2022-12-20 23:08:29,069] [INFO] [timer.py:197:stop] 0/7572, RunningAvgSamplesPerSec=5.878446125690778, CurrSamplesPerSec=5.330949789403662, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3741/5000 [12:46:09<3:51:55, 11.05s/it][2022-12-20 23:08:40,066] [INFO] [timer.py:197:stop] 0/7574, RunningAvgSamplesPerSec=5.8784479987905955, CurrSamplesPerSec=5.3168972325243855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3742/5000 [12:46:20<3:51:38, 11.05s/it][2022-12-20 23:08:51,185] [INFO] [timer.py:197:stop] 0/7576, RunningAvgSamplesPerSec=5.878432517080829, CurrSamplesPerSec=5.245709404510057, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3743/5000 [12:46:31<3:51:44, 11.06s/it][2022-12-20 23:09:02,232] [INFO] [timer.py:197:stop] 0/7578, RunningAvgSamplesPerSec=5.878427278657847, CurrSamplesPerSec=5.285034550385853, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3744/5000 [12:46:42<3:51:45, 11.07s/it][2022-12-20 23:09:13,275] [INFO] [logging.py:68:log_dist] [Rank 0] step=3790, skipped=5, lr=[2.702222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:09:13,276] [INFO] [timer.py:197:stop] 0/7580, RunningAvgSamplesPerSec=5.878422344671622, CurrSamplesPerSec=5.3279310537696265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3745/5000 [12:46:53<3:51:09, 11.05s/it][2022-12-20 23:09:24,247] [INFO] [timer.py:197:stop] 0/7582, RunningAvgSamplesPerSec=5.878428146982947, CurrSamplesPerSec=5.359324300031728, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3746/5000 [12:47:04<3:50:33, 11.03s/it][2022-12-20 23:09:35,282] [INFO] [timer.py:197:stop] 0/7584, RunningAvgSamplesPerSec=5.878424526920814, CurrSamplesPerSec=5.313822225130846, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3747/5000 [12:47:15<3:50:15, 11.03s/it][2022-12-20 23:09:46,235] [INFO] [timer.py:197:stop] 0/7586, RunningAvgSamplesPerSec=5.878432695988677, CurrSamplesPerSec=5.366853476741881, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3748/5000 [12:47:26<3:49:37, 11.00s/it][2022-12-20 23:09:57,265] [INFO] [timer.py:197:stop] 0/7588, RunningAvgSamplesPerSec=5.878429971909111, CurrSamplesPerSec=5.299998456016633, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▍  | 3749/5000 [12:47:37<3:49:52, 11.02s/it][2022-12-20 23:10:08,345] [INFO] [timer.py:197:stop] 0/7590, RunningAvgSamplesPerSec=5.878420082765468, CurrSamplesPerSec=5.2934155768350974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3750/5000 [12:47:48<3:49:44, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.6911111111111114e-06, 'epoch': 91.46}

 75%|███████▌  | 3750/5000 [12:47:48<3:49:44, 11.03s/it][2022-12-20 23:10:19,354] [INFO] [timer.py:197:stop] 0/7592, RunningAvgSamplesPerSec=5.878420417573296, CurrSamplesPerSec=5.316504870059522, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3751/5000 [12:47:59<3:49:38, 11.03s/it][2022-12-20 23:10:30,401] [INFO] [timer.py:197:stop] 0/7594, RunningAvgSamplesPerSec=5.878415245318523, CurrSamplesPerSec=5.310415835704934, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3752/5000 [12:48:10<3:49:36, 11.04s/it][2022-12-20 23:10:41,412] [INFO] [timer.py:197:stop] 0/7596, RunningAvgSamplesPerSec=5.8784152978270185, CurrSamplesPerSec=5.351687542602, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3753/5000 [12:48:21<3:49:06, 11.02s/it][2022-12-20 23:10:52,414] [INFO] [timer.py:197:stop] 0/7598, RunningAvgSamplesPerSec=5.878416438312663, CurrSamplesPerSec=5.337076693404328, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3754/5000 [12:48:32<3:48:41, 11.01s/it][2022-12-20 23:11:03,443] [INFO] [logging.py:68:log_dist] [Rank 0] step=3800, skipped=5, lr=[2.68e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:11:03,445] [INFO] [timer.py:197:stop] 0/7600, RunningAvgSamplesPerSec=5.878413575259489, CurrSamplesPerSec=5.29718985280888, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3755/5000 [12:48:44<3:48:54, 11.03s/it][2022-12-20 23:11:14,490] [INFO] [timer.py:197:stop] 0/7602, RunningAvgSamplesPerSec=5.878408840050538, CurrSamplesPerSec=5.328150175236068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3756/5000 [12:48:54<3:48:22, 11.01s/it][2022-12-20 23:11:25,484] [INFO] [timer.py:197:stop] 0/7604, RunningAvgSamplesPerSec=5.878411182779143, CurrSamplesPerSec=5.309906577373378, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3757/5000 [12:49:06<3:48:10, 11.01s/it][2022-12-20 23:11:36,437] [INFO] [timer.py:197:stop] 0/7606, RunningAvgSamplesPerSec=5.878419213635451, CurrSamplesPerSec=5.363168537846921, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3758/5000 [12:49:16<3:47:50, 11.01s/it][2022-12-20 23:11:47,469] [INFO] [timer.py:197:stop] 0/7608, RunningAvgSamplesPerSec=5.878416188182511, CurrSamplesPerSec=5.325658837249563, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3759/5000 [12:49:28<3:47:42, 11.01s/it][2022-12-20 23:11:58,487] [INFO] [timer.py:197:stop] 0/7610, RunningAvgSamplesPerSec=5.878415005700013, CurrSamplesPerSec=5.323341896885336, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3760/5000 [12:49:38<3:47:17, 11.00s/it][2022-12-20 23:12:09,419] [INFO] [timer.py:197:stop] 0/7612, RunningAvgSamplesPerSec=5.878426013345752, CurrSamplesPerSec=5.358954322625611, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3761/5000 [12:49:49<3:46:44, 10.98s/it][2022-12-20 23:12:20,367] [INFO] [timer.py:197:stop] 0/7614, RunningAvgSamplesPerSec=5.878434786467438, CurrSamplesPerSec=5.350085898992479, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3762/5000 [12:50:00<3:46:36, 10.98s/it][2022-12-20 23:12:31,431] [INFO] [timer.py:197:stop] 0/7616, RunningAvgSamplesPerSec=5.878427200673542, CurrSamplesPerSec=5.283833424233183, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3763/5000 [12:50:11<3:46:38, 10.99s/it][2022-12-20 23:12:42,413] [INFO] [timer.py:197:stop] 0/7618, RunningAvgSamplesPerSec=5.878431104278507, CurrSamplesPerSec=5.3149065337550585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3764/5000 [12:50:22<3:46:42, 11.01s/it][2022-12-20 23:12:53,406] [INFO] [logging.py:68:log_dist] [Rank 0] step=3810, skipped=5, lr=[2.6577777777777782e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:12:53,408] [INFO] [timer.py:197:stop] 0/7620, RunningAvgSamplesPerSec=5.878433118504083, CurrSamplesPerSec=5.348069848839359, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3765/5000 [12:50:33<3:46:27, 11.00s/it][2022-12-20 23:13:04,438] [INFO] [timer.py:197:stop] 0/7622, RunningAvgSamplesPerSec=5.87843023776621, CurrSamplesPerSec=5.316054030116023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3766/5000 [12:50:44<3:46:26, 11.01s/it][2022-12-20 23:13:15,462] [INFO] [timer.py:197:stop] 0/7624, RunningAvgSamplesPerSec=5.878428356265862, CurrSamplesPerSec=5.321885046605942, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3767/5000 [12:50:56<3:46:19, 11.01s/it][2022-12-20 23:13:26,454] [INFO] [timer.py:197:stop] 0/7626, RunningAvgSamplesPerSec=5.878430907917786, CurrSamplesPerSec=5.347921535076297, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3768/5000 [12:51:07<3:46:06, 11.01s/it][2022-12-20 23:13:37,472] [INFO] [timer.py:197:stop] 0/7628, RunningAvgSamplesPerSec=5.878429871828313, CurrSamplesPerSec=5.339071294555368, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3769/5000 [12:51:17<3:45:44, 11.00s/it][2022-12-20 23:13:48,456] [INFO] [timer.py:197:stop] 0/7630, RunningAvgSamplesPerSec=5.878433763802678, CurrSamplesPerSec=5.33728871482736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3770/5000 [12:51:28<3:45:25, 11.00s/it][2022-12-20 23:13:59,442] [INFO] [timer.py:197:stop] 0/7632, RunningAvgSamplesPerSec=5.878437055470631, CurrSamplesPerSec=5.333166339593109, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3771/5000 [12:51:39<3:45:22, 11.00s/it][2022-12-20 23:14:10,435] [INFO] [timer.py:197:stop] 0/7634, RunningAvgSamplesPerSec=5.87843933398549, CurrSamplesPerSec=5.355107147844475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3772/5000 [12:51:50<3:45:06, 11.00s/it][2022-12-20 23:14:20,533] [INFO] [timer.py:197:stop] 0/7636, RunningAvgSamplesPerSec=5.878566986777598, CurrSamplesPerSec=6.280583818739152, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 23:14:31,586] [INFO] [timer.py:197:stop] 0/7638, RunningAvgSamplesPerSec=5.878560216601736, CurrSamplesPerSec=5.2951411818922045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3773/5000 [12:52:07<4:16:50, 12.56s/it][2022-12-20 23:14:42,540] [INFO] [logging.py:68:log_dist] [Rank 0] step=3820, skipped=5, lr=[2.635555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:14:42,542] [INFO] [timer.py:197:stop] 0/7640, RunningAvgSamplesPerSec=5.878567919694184, CurrSamplesPerSec=5.342437576369452, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 75%|███████▌  | 3774/5000 [12:52:18<4:06:47, 12.08s/it][2022-12-20 23:14:53,485] [INFO] [timer.py:197:stop] 0/7642, RunningAvgSamplesPerSec=5.878577431240252, CurrSamplesPerSec=5.327569627337699, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3775/5000 [12:52:29<3:59:38, 11.74s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.6333333333333332e-06, 'epoch': 92.07}

 76%|███████▌  | 3775/5000 [12:52:29<3:59:38, 11.74s/it][2022-12-20 23:15:04,433] [INFO] [timer.py:197:stop] 0/7644, RunningAvgSamplesPerSec=5.878586291757545, CurrSamplesPerSec=5.355597973489142, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3776/5000 [12:52:40<3:54:37, 11.50s/it][2022-12-20 23:15:15,445] [INFO] [timer.py:197:stop] 0/7646, RunningAvgSamplesPerSec=5.878585997334733, CurrSamplesPerSec=5.324745888049905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3777/5000 [12:52:51<3:51:26, 11.35s/it][2022-12-20 23:15:26,396] [INFO] [timer.py:197:stop] 0/7648, RunningAvgSamplesPerSec=5.878594537053175, CurrSamplesPerSec=5.351348489285001, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3778/5000 [12:53:01<3:48:46, 11.23s/it][2022-12-20 23:15:37,428] [INFO] [timer.py:197:stop] 0/7650, RunningAvgSamplesPerSec=5.878591396775819, CurrSamplesPerSec=5.333393521458311, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3779/5000 [12:53:13<3:47:22, 11.17s/it][2022-12-20 23:15:48,394] [INFO] [timer.py:197:stop] 0/7652, RunningAvgSamplesPerSec=5.87859779128862, CurrSamplesPerSec=5.337131448074324, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3780/5000 [12:53:23<3:45:55, 11.11s/it][2022-12-20 23:15:59,371] [INFO] [timer.py:197:stop] 0/7654, RunningAvgSamplesPerSec=5.878602394097781, CurrSamplesPerSec=5.3624134321341375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3781/5000 [12:53:34<3:44:55, 11.07s/it][2022-12-20 23:16:10,378] [INFO] [timer.py:197:stop] 0/7656, RunningAvgSamplesPerSec=5.878602919379653, CurrSamplesPerSec=5.300531981483919, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3782/5000 [12:53:45<3:44:20, 11.05s/it][2022-12-20 23:16:21,383] [INFO] [timer.py:197:stop] 0/7658, RunningAvgSamplesPerSec=5.878603912729005, CurrSamplesPerSec=5.337015148798557, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3783/5000 [12:53:56<3:43:52, 11.04s/it][2022-12-20 23:16:32,457] [INFO] [logging.py:68:log_dist] [Rank 0] step=3830, skipped=5, lr=[2.6133333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:16:32,458] [INFO] [timer.py:197:stop] 0/7660, RunningAvgSamplesPerSec=5.8785949973467995, CurrSamplesPerSec=5.294370232244625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3784/5000 [12:54:08<3:43:55, 11.05s/it][2022-12-20 23:16:43,469] [INFO] [timer.py:197:stop] 0/7662, RunningAvgSamplesPerSec=5.878594951985115, CurrSamplesPerSec=5.350608866302717, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3785/5000 [12:54:19<3:43:30, 11.04s/it][2022-12-20 23:16:54,510] [INFO] [timer.py:197:stop] 0/7664, RunningAvgSamplesPerSec=5.87859055680708, CurrSamplesPerSec=5.326493674790825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3786/5000 [12:54:30<3:43:20, 11.04s/it][2022-12-20 23:17:05,507] [INFO] [timer.py:197:stop] 0/7666, RunningAvgSamplesPerSec=5.878592709841305, CurrSamplesPerSec=5.3628333852298065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3787/5000 [12:54:41<3:42:54, 11.03s/it][2022-12-20 23:17:16,476] [INFO] [timer.py:197:stop] 0/7668, RunningAvgSamplesPerSec=5.878598451275768, CurrSamplesPerSec=5.372889657985085, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3788/5000 [12:54:52<3:42:22, 11.01s/it][2022-12-20 23:17:27,469] [INFO] [timer.py:197:stop] 0/7670, RunningAvgSamplesPerSec=5.878600911663718, CurrSamplesPerSec=5.335066404654903, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3789/5000 [12:55:03<3:42:05, 11.00s/it][2022-12-20 23:17:38,463] [INFO] [timer.py:197:stop] 0/7672, RunningAvgSamplesPerSec=5.878602977237013, CurrSamplesPerSec=5.331597363599247, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3790/5000 [12:55:14<3:41:51, 11.00s/it][2022-12-20 23:17:49,524] [INFO] [timer.py:197:stop] 0/7674, RunningAvgSamplesPerSec=5.87859584127065, CurrSamplesPerSec=5.3280801643912445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3791/5000 [12:55:25<3:42:02, 11.02s/it][2022-12-20 23:18:00,485] [INFO] [timer.py:197:stop] 0/7676, RunningAvgSamplesPerSec=5.878602697648732, CurrSamplesPerSec=5.349374554584891, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3792/5000 [12:55:36<3:41:30, 11.00s/it][2022-12-20 23:18:11,535] [INFO] [timer.py:197:stop] 0/7678, RunningAvgSamplesPerSec=5.8785971077164065, CurrSamplesPerSec=5.298085220168338, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3793/5000 [12:55:47<3:41:36, 11.02s/it][2022-12-20 23:18:22,570] [INFO] [logging.py:68:log_dist] [Rank 0] step=3840, skipped=5, lr=[2.5911111111111115e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:18:22,571] [INFO] [timer.py:197:stop] 0/7680, RunningAvgSamplesPerSec=5.878593255098724, CurrSamplesPerSec=5.295094387976816, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3794/5000 [12:55:58<3:41:32, 11.02s/it][2022-12-20 23:18:33,528] [INFO] [timer.py:197:stop] 0/7682, RunningAvgSamplesPerSec=5.878600691072159, CurrSamplesPerSec=5.318931790460341, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3795/5000 [12:56:09<3:40:58, 11.00s/it][2022-12-20 23:18:44,593] [INFO] [timer.py:197:stop] 0/7684, RunningAvgSamplesPerSec=5.878593226341898, CurrSamplesPerSec=5.301164858077294, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3796/5000 [12:56:20<3:41:09, 11.02s/it][2022-12-20 23:18:55,639] [INFO] [timer.py:197:stop] 0/7686, RunningAvgSamplesPerSec=5.8785882053688, CurrSamplesPerSec=5.315990021675839, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3797/5000 [12:56:31<3:41:07, 11.03s/it][2022-12-20 23:19:06,623] [INFO] [timer.py:197:stop] 0/7688, RunningAvgSamplesPerSec=5.878591872748984, CurrSamplesPerSec=5.364671024826193, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3798/5000 [12:56:42<3:40:40, 11.02s/it][2022-12-20 23:19:17,650] [INFO] [timer.py:197:stop] 0/7690, RunningAvgSamplesPerSec=5.878589557902313, CurrSamplesPerSec=5.323912865325096, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3799/5000 [12:56:53<3:40:33, 11.02s/it][2022-12-20 23:19:28,666] [INFO] [timer.py:197:stop] 0/7692, RunningAvgSamplesPerSec=5.878588644239332, CurrSamplesPerSec=5.352910326346408, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3800/5000 [12:57:04<3:40:21, 11.02s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.577777777777778e-06, 'epoch': 92.67}

 76%|███████▌  | 3800/5000 [12:57:04<3:40:21, 11.02s/it][2022-12-20 23:19:39,618] [INFO] [timer.py:197:stop] 0/7694, RunningAvgSamplesPerSec=5.878596966314001, CurrSamplesPerSec=5.351007985583689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3801/5000 [12:57:15<3:39:46, 11.00s/it][2022-12-20 23:19:50,671] [INFO] [timer.py:197:stop] 0/7696, RunningAvgSamplesPerSec=5.87859079335074, CurrSamplesPerSec=5.315875694646052, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3802/5000 [12:57:26<3:39:55, 11.01s/it][2022-12-20 23:20:01,730] [INFO] [timer.py:197:stop] 0/7698, RunningAvgSamplesPerSec=5.87858399454067, CurrSamplesPerSec=5.313520557540341, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3803/5000 [12:57:37<3:40:00, 11.03s/it][2022-12-20 23:20:12,787] [INFO] [logging.py:68:log_dist] [Rank 0] step=3850, skipped=5, lr=[2.568888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:20:12,789] [INFO] [timer.py:197:stop] 0/7700, RunningAvgSamplesPerSec=5.878577034653145, CurrSamplesPerSec=5.301813174166168, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3804/5000 [12:57:48<3:40:00, 11.04s/it][2022-12-20 23:20:23,835] [INFO] [timer.py:197:stop] 0/7702, RunningAvgSamplesPerSec=5.878572037207572, CurrSamplesPerSec=5.306017042952742, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3805/5000 [12:57:59<3:39:52, 11.04s/it][2022-12-20 23:20:34,877] [INFO] [timer.py:197:stop] 0/7704, RunningAvgSamplesPerSec=5.878567540699114, CurrSamplesPerSec=5.306785933117945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3806/5000 [12:58:10<3:39:42, 11.04s/it][2022-12-20 23:20:45,901] [INFO] [timer.py:197:stop] 0/7706, RunningAvgSamplesPerSec=5.8785655929271226, CurrSamplesPerSec=5.307793694881325, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3807/5000 [12:58:21<3:39:25, 11.04s/it][2022-12-20 23:20:56,898] [INFO] [timer.py:197:stop] 0/7708, RunningAvgSamplesPerSec=5.8785676051345215, CurrSamplesPerSec=5.332346780832149, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3808/5000 [12:58:32<3:39:00, 11.02s/it][2022-12-20 23:21:07,931] [INFO] [timer.py:197:stop] 0/7710, RunningAvgSamplesPerSec=5.878564400617195, CurrSamplesPerSec=5.286267244907392, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3809/5000 [12:58:43<3:38:52, 11.03s/it][2022-12-20 23:21:18,951] [INFO] [timer.py:197:stop] 0/7712, RunningAvgSamplesPerSec=5.878562904670783, CurrSamplesPerSec=5.31072702717546, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3810/5000 [12:58:54<3:38:39, 11.02s/it][2022-12-20 23:21:29,927] [INFO] [timer.py:197:stop] 0/7714, RunningAvgSamplesPerSec=5.878567572074026, CurrSamplesPerSec=5.3486646781985705, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3811/5000 [12:59:05<3:38:11, 11.01s/it][2022-12-20 23:21:40,935] [INFO] [timer.py:197:stop] 0/7716, RunningAvgSamplesPerSec=5.878567933335444, CurrSamplesPerSec=5.311430231469416, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▌  | 3812/5000 [12:59:16<3:37:59, 11.01s/it][2022-12-20 23:21:51,961] [INFO] [timer.py:197:stop] 0/7718, RunningAvgSamplesPerSec=5.87856563588244, CurrSamplesPerSec=5.319589309021926, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3813/5000 [12:59:27<3:37:54, 11.01s/it][2022-12-20 23:22:02,069] [INFO] [logging.py:68:log_dist] [Rank 0] step=3860, skipped=5, lr=[2.5466666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:22:02,070] [INFO] [timer.py:197:stop] 0/7720, RunningAvgSamplesPerSec=5.878690290838976, CurrSamplesPerSec=5.313593762427347, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3814/5000 [12:59:42<4:01:37, 12.22s/it][2022-12-20 23:22:13,103] [INFO] [timer.py:197:stop] 0/7722, RunningAvgSamplesPerSec=5.878687284216578, CurrSamplesPerSec=5.305649774620705, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3815/5000 [12:59:53<3:54:34, 11.88s/it][2022-12-20 23:22:24,089] [INFO] [timer.py:197:stop] 0/7724, RunningAvgSamplesPerSec=5.878690616464378, CurrSamplesPerSec=5.378639163623882, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3816/5000 [13:00:04<3:48:58, 11.60s/it][2022-12-20 23:22:35,103] [INFO] [timer.py:197:stop] 0/7726, RunningAvgSamplesPerSec=5.878690063750731, CurrSamplesPerSec=5.335016569756026, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3817/5000 [13:00:15<3:45:24, 11.43s/it][2022-12-20 23:22:46,178] [INFO] [timer.py:197:stop] 0/7728, RunningAvgSamplesPerSec=5.878680968754474, CurrSamplesPerSec=5.296797466997161, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3818/5000 [13:00:26<3:42:59, 11.32s/it][2022-12-20 23:22:57,166] [INFO] [timer.py:197:stop] 0/7730, RunningAvgSamplesPerSec=5.878684206717754, CurrSamplesPerSec=5.357013046275805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3819/5000 [13:00:37<3:40:50, 11.22s/it][2022-12-20 23:23:08,179] [INFO] [timer.py:197:stop] 0/7732, RunningAvgSamplesPerSec=5.8786838219750726, CurrSamplesPerSec=5.334283786157653, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3820/5000 [13:00:48<3:39:11, 11.15s/it][2022-12-20 23:23:19,134] [INFO] [timer.py:197:stop] 0/7734, RunningAvgSamplesPerSec=5.87869146120634, CurrSamplesPerSec=5.349372848950356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3821/5000 [13:00:59<3:38:01, 11.10s/it][2022-12-20 23:23:30,149] [INFO] [timer.py:197:stop] 0/7736, RunningAvgSamplesPerSec=5.878691012062508, CurrSamplesPerSec=5.317719001456355, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3822/5000 [13:01:10<3:37:32, 11.08s/it][2022-12-20 23:23:41,214] [INFO] [timer.py:197:stop] 0/7738, RunningAvgSamplesPerSec=5.8786832082061045, CurrSamplesPerSec=5.298355646320129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3823/5000 [13:01:21<3:37:00, 11.06s/it][2022-12-20 23:23:52,179] [INFO] [logging.py:68:log_dist] [Rank 0] step=3870, skipped=5, lr=[2.5244444444444447e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:23:52,181] [INFO] [timer.py:197:stop] 0/7740, RunningAvgSamplesPerSec=5.878689034021425, CurrSamplesPerSec=5.346106636517864, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3824/5000 [13:01:32<3:36:25, 11.04s/it][2022-12-20 23:24:03,180] [INFO] [timer.py:197:stop] 0/7742, RunningAvgSamplesPerSec=5.878690594964675, CurrSamplesPerSec=5.342014858073612, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 76%|███████▋  | 3825/5000 [13:01:43<3:35:54, 11.02s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.5222222222222226e-06, 'epoch': 93.29}

 76%|███████▋  | 3825/5000 [13:01:43<3:35:54, 11.02s/it][2022-12-20 23:24:14,158] [INFO] [timer.py:197:stop] 0/7744, RunningAvgSamplesPerSec=5.878695382177821, CurrSamplesPerSec=5.349904633206649, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3826/5000 [13:01:54<3:35:24, 11.01s/it][2022-12-20 23:24:25,188] [INFO] [timer.py:197:stop] 0/7746, RunningAvgSamplesPerSec=5.878692574073494, CurrSamplesPerSec=5.297778854210213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3827/5000 [13:02:05<3:35:29, 11.02s/it][2022-12-20 23:24:36,199] [INFO] [timer.py:197:stop] 0/7748, RunningAvgSamplesPerSec=5.878692494612233, CurrSamplesPerSec=5.335400852724543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3828/5000 [13:02:16<3:35:05, 11.01s/it][2022-12-20 23:24:47,249] [INFO] [timer.py:197:stop] 0/7750, RunningAvgSamplesPerSec=5.878686802274691, CurrSamplesPerSec=5.277901349469991, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3829/5000 [13:02:27<3:34:59, 11.02s/it][2022-12-20 23:24:58,221] [INFO] [timer.py:197:stop] 0/7752, RunningAvgSamplesPerSec=5.878691952741859, CurrSamplesPerSec=5.32447783074356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3830/5000 [13:02:38<3:34:59, 11.03s/it][2022-12-20 23:25:09,236] [INFO] [timer.py:197:stop] 0/7754, RunningAvgSamplesPerSec=5.878691361289831, CurrSamplesPerSec=5.3540691674313425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3831/5000 [13:02:49<3:34:27, 11.01s/it][2022-12-20 23:25:20,199] [INFO] [timer.py:197:stop] 0/7756, RunningAvgSamplesPerSec=5.878697798645662, CurrSamplesPerSec=5.355799714599917, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3832/5000 [13:03:00<3:34:09, 11.00s/it][2022-12-20 23:25:31,229] [INFO] [timer.py:197:stop] 0/7758, RunningAvgSamplesPerSec=5.878694874581168, CurrSamplesPerSec=5.318756844964015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3833/5000 [13:03:11<3:34:12, 11.01s/it][2022-12-20 23:25:42,266] [INFO] [logging.py:68:log_dist] [Rank 0] step=3880, skipped=5, lr=[2.5022222222222224e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:25:42,268] [INFO] [timer.py:197:stop] 0/7760, RunningAvgSamplesPerSec=5.878690638549906, CurrSamplesPerSec=5.320031470449167, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3834/5000 [13:03:22<3:34:03, 11.01s/it][2022-12-20 23:25:53,227] [INFO] [timer.py:197:stop] 0/7762, RunningAvgSamplesPerSec=5.878697615316082, CurrSamplesPerSec=5.372930308967938, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3835/5000 [13:03:33<3:33:35, 11.00s/it][2022-12-20 23:26:04,240] [INFO] [timer.py:197:stop] 0/7764, RunningAvgSamplesPerSec=5.878697105599595, CurrSamplesPerSec=5.331084034829067, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3836/5000 [13:03:44<3:33:34, 11.01s/it][2022-12-20 23:26:15,255] [INFO] [timer.py:197:stop] 0/7766, RunningAvgSamplesPerSec=5.878696336007531, CurrSamplesPerSec=5.343238330338344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3837/5000 [13:03:55<3:33:03, 10.99s/it][2022-12-20 23:26:26,225] [INFO] [timer.py:197:stop] 0/7768, RunningAvgSamplesPerSec=5.878701818308288, CurrSamplesPerSec=5.326377204529283, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3838/5000 [13:04:06<3:32:53, 10.99s/it][2022-12-20 23:26:37,210] [INFO] [timer.py:197:stop] 0/7770, RunningAvgSamplesPerSec=5.878705273640628, CurrSamplesPerSec=5.335321955678633, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3839/5000 [13:04:17<3:32:51, 11.00s/it][2022-12-20 23:26:48,231] [INFO] [timer.py:197:stop] 0/7772, RunningAvgSamplesPerSec=5.87870380643178, CurrSamplesPerSec=5.333633439737585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3840/5000 [13:04:28<3:32:51, 11.01s/it][2022-12-20 23:26:59,242] [INFO] [timer.py:197:stop] 0/7774, RunningAvgSamplesPerSec=5.878703564924367, CurrSamplesPerSec=5.352393952753218, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3841/5000 [13:04:39<3:32:33, 11.00s/it][2022-12-20 23:27:10,239] [INFO] [timer.py:197:stop] 0/7776, RunningAvgSamplesPerSec=5.878705403333957, CurrSamplesPerSec=5.346318524168043, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3842/5000 [13:04:50<3:32:21, 11.00s/it][2022-12-20 23:27:21,233] [INFO] [timer.py:197:stop] 0/7778, RunningAvgSamplesPerSec=5.878707602723294, CurrSamplesPerSec=5.351187619354196, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3843/5000 [13:05:01<3:32:07, 11.00s/it][2022-12-20 23:27:32,210] [INFO] [logging.py:68:log_dist] [Rank 0] step=3890, skipped=5, lr=[2.4800000000000004e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:27:32,211] [INFO] [timer.py:197:stop] 0/7780, RunningAvgSamplesPerSec=5.878711716009359, CurrSamplesPerSec=5.365132293602501, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3844/5000 [13:05:12<3:31:44, 10.99s/it][2022-12-20 23:27:43,200] [INFO] [timer.py:197:stop] 0/7782, RunningAvgSamplesPerSec=5.878714437552027, CurrSamplesPerSec=5.345178787747728, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3845/5000 [13:05:23<3:31:34, 10.99s/it][2022-12-20 23:27:54,211] [INFO] [timer.py:197:stop] 0/7784, RunningAvgSamplesPerSec=5.878714088006636, CurrSamplesPerSec=5.327882621051407, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3846/5000 [13:05:34<3:31:32, 11.00s/it][2022-12-20 23:28:05,240] [INFO] [timer.py:197:stop] 0/7786, RunningAvgSamplesPerSec=5.878711294867211, CurrSamplesPerSec=5.319041611524896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3847/5000 [13:05:45<3:31:31, 11.01s/it][2022-12-20 23:28:16,268] [INFO] [timer.py:197:stop] 0/7788, RunningAvgSamplesPerSec=5.878708684225889, CurrSamplesPerSec=5.317142198841507, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3848/5000 [13:05:56<3:31:25, 11.01s/it][2022-12-20 23:28:27,289] [INFO] [timer.py:197:stop] 0/7790, RunningAvgSamplesPerSec=5.878707173442277, CurrSamplesPerSec=5.319617561297324, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3849/5000 [13:06:07<3:31:25, 11.02s/it][2022-12-20 23:28:38,333] [INFO] [timer.py:197:stop] 0/7792, RunningAvgSamplesPerSec=5.878702397662285, CurrSamplesPerSec=5.317858270219842, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3850/5000 [13:06:18<3:30:51, 11.00s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.466666666666667e-06, 'epoch': 93.89}

 77%|███████▋  | 3850/5000 [13:06:18<3:30:51, 11.00s/it][2022-12-20 23:28:49,284] [INFO] [timer.py:197:stop] 0/7794, RunningAvgSamplesPerSec=5.8787110497235275, CurrSamplesPerSec=5.323519044476014, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3851/5000 [13:06:29<3:30:51, 11.01s/it][2022-12-20 23:29:00,258] [INFO] [timer.py:197:stop] 0/7796, RunningAvgSamplesPerSec=5.878716097044405, CurrSamplesPerSec=5.375528300180477, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3852/5000 [13:06:40<3:30:24, 11.00s/it][2022-12-20 23:29:11,282] [INFO] [timer.py:197:stop] 0/7798, RunningAvgSamplesPerSec=5.878714039258267, CurrSamplesPerSec=5.321443420961807, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3853/5000 [13:06:51<3:30:23, 11.01s/it][2022-12-20 23:29:22,301] [INFO] [logging.py:68:log_dist] [Rank 0] step=3900, skipped=5, lr=[2.457777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:29:22,302] [INFO] [timer.py:197:stop] 0/7800, RunningAvgSamplesPerSec=5.878712487563296, CurrSamplesPerSec=5.327456281928265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3854/5000 [13:07:02<3:30:16, 11.01s/it][2022-12-20 23:29:32,394] [INFO] [timer.py:197:stop] 0/7802, RunningAvgSamplesPerSec=5.878838202380105, CurrSamplesPerSec=6.283705344006855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 23:29:43,397] [INFO] [timer.py:197:stop] 0/7804, RunningAvgSamplesPerSec=5.878838781812679, CurrSamplesPerSec=5.316228166140369, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3855/5000 [13:07:18<3:59:30, 12.55s/it][2022-12-20 23:29:54,493] [INFO] [timer.py:197:stop] 0/7806, RunningAvgSamplesPerSec=5.878826764871694, CurrSamplesPerSec=5.261058189103867, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3856/5000 [13:07:30<3:50:59, 12.11s/it][2022-12-20 23:30:05,472] [INFO] [timer.py:197:stop] 0/7808, RunningAvgSamplesPerSec=5.87883102743074, CurrSamplesPerSec=5.327462414299021, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3857/5000 [13:07:41<3:44:17, 11.77s/it][2022-12-20 23:30:16,475] [INFO] [timer.py:197:stop] 0/7810, RunningAvgSamplesPerSec=5.878832168099429, CurrSamplesPerSec=5.344308290977387, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3858/5000 [13:07:52<3:39:41, 11.54s/it][2022-12-20 23:30:27,524] [INFO] [timer.py:197:stop] 0/7812, RunningAvgSamplesPerSec=5.878826658369451, CurrSamplesPerSec=5.312717329520715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3859/5000 [13:08:03<3:36:41, 11.39s/it][2022-12-20 23:30:38,500] [INFO] [timer.py:197:stop] 0/7814, RunningAvgSamplesPerSec=5.878831396450725, CurrSamplesPerSec=5.34191599247152, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3860/5000 [13:08:14<3:34:06, 11.27s/it][2022-12-20 23:30:49,529] [INFO] [timer.py:197:stop] 0/7816, RunningAvgSamplesPerSec=5.87882868387825, CurrSamplesPerSec=5.313649298514762, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3861/5000 [13:08:25<3:32:33, 11.20s/it][2022-12-20 23:31:00,534] [INFO] [timer.py:197:stop] 0/7818, RunningAvgSamplesPerSec=5.878829414144464, CurrSamplesPerSec=5.306470377972026, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3862/5000 [13:08:36<3:31:16, 11.14s/it][2022-12-20 23:31:11,499] [INFO] [logging.py:68:log_dist] [Rank 0] step=3910, skipped=5, lr=[2.4355555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:31:11,500] [INFO] [timer.py:197:stop] 0/7820, RunningAvgSamplesPerSec=5.878835327409504, CurrSamplesPerSec=5.340538208194829, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3863/5000 [13:08:47<3:30:06, 11.09s/it][2022-12-20 23:31:22,535] [INFO] [timer.py:197:stop] 0/7822, RunningAvgSamplesPerSec=5.878831790766284, CurrSamplesPerSec=5.309960145744297, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3864/5000 [13:08:58<3:29:37, 11.07s/it][2022-12-20 23:31:33,531] [INFO] [timer.py:197:stop] 0/7824, RunningAvgSamplesPerSec=5.878833681674794, CurrSamplesPerSec=5.330578638109024, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3865/5000 [13:09:09<3:29:00, 11.05s/it][2022-12-20 23:31:44,540] [INFO] [timer.py:197:stop] 0/7826, RunningAvgSamplesPerSec=5.87883370143866, CurrSamplesPerSec=5.3376796933919985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3866/5000 [13:09:20<3:28:36, 11.04s/it][2022-12-20 23:31:55,470] [INFO] [timer.py:197:stop] 0/7828, RunningAvgSamplesPerSec=5.8788447908938055, CurrSamplesPerSec=5.338036336169464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3867/5000 [13:09:31<3:27:48, 11.00s/it][2022-12-20 23:32:06,545] [INFO] [timer.py:197:stop] 0/7830, RunningAvgSamplesPerSec=5.878836551761894, CurrSamplesPerSec=5.297575396604553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3868/5000 [13:09:42<3:28:01, 11.03s/it][2022-12-20 23:32:17,566] [INFO] [timer.py:197:stop] 0/7832, RunningAvgSamplesPerSec=5.878835123760748, CurrSamplesPerSec=5.326752210468932, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3869/5000 [13:09:53<3:27:48, 11.02s/it][2022-12-20 23:32:28,548] [INFO] [timer.py:197:stop] 0/7834, RunningAvgSamplesPerSec=5.878838960687376, CurrSamplesPerSec=5.306401355286229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3870/5000 [13:10:04<3:27:23, 11.01s/it][2022-12-20 23:32:39,566] [INFO] [timer.py:197:stop] 0/7836, RunningAvgSamplesPerSec=5.878837959573247, CurrSamplesPerSec=5.328541085307449, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3871/5000 [13:10:15<3:27:14, 11.01s/it][2022-12-20 23:32:50,596] [INFO] [timer.py:197:stop] 0/7838, RunningAvgSamplesPerSec=5.878835123463715, CurrSamplesPerSec=5.33660177561313, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3872/5000 [13:10:26<3:27:08, 11.02s/it][2022-12-20 23:33:01,627] [INFO] [logging.py:68:log_dist] [Rank 0] step=3920, skipped=5, lr=[2.4133333333333337e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:33:01,628] [INFO] [timer.py:197:stop] 0/7840, RunningAvgSamplesPerSec=5.87883185203176, CurrSamplesPerSec=5.294645292011416, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3873/5000 [13:10:37<3:27:02, 11.02s/it][2022-12-20 23:33:12,661] [INFO] [timer.py:197:stop] 0/7842, RunningAvgSamplesPerSec=5.878828715323466, CurrSamplesPerSec=5.312198378624068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 77%|███████▋  | 3874/5000 [13:10:48<3:26:54, 11.03s/it][2022-12-20 23:33:23,648] [INFO] [timer.py:197:stop] 0/7844, RunningAvgSamplesPerSec=5.878831850766411, CurrSamplesPerSec=5.359287706583335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3875/5000 [13:10:59<3:26:30, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.4088888888888894e-06, 'epoch': 94.51}

 78%|███████▊  | 3875/5000 [13:10:59<3:26:30, 11.01s/it][2022-12-20 23:33:34,675] [INFO] [timer.py:197:stop] 0/7846, RunningAvgSamplesPerSec=5.878829898324588, CurrSamplesPerSec=5.311153845347203, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3876/5000 [13:11:10<3:26:24, 11.02s/it][2022-12-20 23:33:45,681] [INFO] [timer.py:197:stop] 0/7848, RunningAvgSamplesPerSec=5.878830387191428, CurrSamplesPerSec=5.36021275396794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3877/5000 [13:11:21<3:26:09, 11.01s/it][2022-12-20 23:33:56,726] [INFO] [timer.py:197:stop] 0/7850, RunningAvgSamplesPerSec=5.878825538888365, CurrSamplesPerSec=5.30296864198283, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3878/5000 [13:11:32<3:26:08, 11.02s/it][2022-12-20 23:34:07,731] [INFO] [timer.py:197:stop] 0/7852, RunningAvgSamplesPerSec=5.878826005293767, CurrSamplesPerSec=5.337938891536074, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3879/5000 [13:11:43<3:25:51, 11.02s/it][2022-12-20 23:34:18,783] [INFO] [timer.py:197:stop] 0/7854, RunningAvgSamplesPerSec=5.878820233112816, CurrSamplesPerSec=5.316866271062725, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3880/5000 [13:11:54<3:25:51, 11.03s/it][2022-12-20 23:34:29,770] [INFO] [timer.py:197:stop] 0/7856, RunningAvgSamplesPerSec=5.878823313426732, CurrSamplesPerSec=5.3257868991281185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3881/5000 [13:12:05<3:25:26, 11.02s/it][2022-12-20 23:34:40,765] [INFO] [timer.py:197:stop] 0/7858, RunningAvgSamplesPerSec=5.878825198992642, CurrSamplesPerSec=5.3520661211071605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3882/5000 [13:12:16<3:25:08, 11.01s/it][2022-12-20 23:34:51,839] [INFO] [logging.py:68:log_dist] [Rank 0] step=3930, skipped=5, lr=[2.3911111111111113e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:34:51,841] [INFO] [timer.py:197:stop] 0/7860, RunningAvgSamplesPerSec=5.878815953332355, CurrSamplesPerSec=5.305035117697726, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3883/5000 [13:12:27<3:25:19, 11.03s/it][2022-12-20 23:35:02,885] [INFO] [timer.py:197:stop] 0/7862, RunningAvgSamplesPerSec=5.878811202483665, CurrSamplesPerSec=5.321673403192416, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3884/5000 [13:12:38<3:25:13, 11.03s/it][2022-12-20 23:35:13,855] [INFO] [timer.py:197:stop] 0/7864, RunningAvgSamplesPerSec=5.87881657635384, CurrSamplesPerSec=5.357456960295431, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3885/5000 [13:12:49<3:24:41, 11.01s/it][2022-12-20 23:35:24,929] [INFO] [timer.py:197:stop] 0/7866, RunningAvgSamplesPerSec=5.878807855383179, CurrSamplesPerSec=5.300307171861124, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3886/5000 [13:13:00<3:24:50, 11.03s/it][2022-12-20 23:35:35,894] [INFO] [timer.py:197:stop] 0/7868, RunningAvgSamplesPerSec=5.878813976675247, CurrSamplesPerSec=5.344225087185568, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3887/5000 [13:13:11<3:24:16, 11.01s/it][2022-12-20 23:35:46,937] [INFO] [timer.py:197:stop] 0/7870, RunningAvgSamplesPerSec=5.878809561246004, CurrSamplesPerSec=5.308943791446665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3888/5000 [13:13:22<3:24:15, 11.02s/it][2022-12-20 23:35:57,998] [INFO] [timer.py:197:stop] 0/7872, RunningAvgSamplesPerSec=5.878802458509132, CurrSamplesPerSec=5.31367117668612, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3889/5000 [13:13:33<3:24:17, 11.03s/it][2022-12-20 23:36:09,049] [INFO] [timer.py:197:stop] 0/7874, RunningAvgSamplesPerSec=5.878796671891111, CurrSamplesPerSec=5.30551722652179, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3890/5000 [13:13:44<3:24:12, 11.04s/it][2022-12-20 23:36:20,110] [INFO] [timer.py:197:stop] 0/7876, RunningAvgSamplesPerSec=5.8787896511832916, CurrSamplesPerSec=5.301572759392883, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3891/5000 [13:13:55<3:24:09, 11.05s/it][2022-12-20 23:36:31,156] [INFO] [timer.py:197:stop] 0/7878, RunningAvgSamplesPerSec=5.878784565907606, CurrSamplesPerSec=5.27618654587394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3892/5000 [13:14:06<3:23:58, 11.05s/it][2022-12-20 23:36:42,194] [INFO] [logging.py:68:log_dist] [Rank 0] step=3940, skipped=5, lr=[2.3688888888888893e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:36:42,195] [INFO] [timer.py:197:stop] 0/7880, RunningAvgSamplesPerSec=5.878780401181073, CurrSamplesPerSec=5.332000641662437, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3893/5000 [13:14:17<3:23:45, 11.04s/it][2022-12-20 23:36:53,223] [INFO] [timer.py:197:stop] 0/7882, RunningAvgSamplesPerSec=5.878778221558751, CurrSamplesPerSec=5.323814668615089, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3894/5000 [13:14:28<3:23:28, 11.04s/it][2022-12-20 23:37:04,167] [INFO] [timer.py:197:stop] 0/7884, RunningAvgSamplesPerSec=5.878787270369022, CurrSamplesPerSec=5.360912423407893, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3895/5000 [13:14:39<3:22:46, 11.01s/it][2022-12-20 23:37:14,263] [INFO] [timer.py:197:stop] 0/7886, RunningAvgSamplesPerSec=5.8789111115507, CurrSamplesPerSec=5.319943749223865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3896/5000 [13:14:54<3:44:56, 12.22s/it][2022-12-20 23:37:25,341] [INFO] [timer.py:197:stop] 0/7888, RunningAvgSamplesPerSec=5.878901803549298, CurrSamplesPerSec=5.28752530250932, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3897/5000 [13:15:05<3:38:22, 11.88s/it][2022-12-20 23:37:36,370] [INFO] [timer.py:197:stop] 0/7890, RunningAvgSamplesPerSec=5.878899109607377, CurrSamplesPerSec=5.326257780239931, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3898/5000 [13:15:16<3:33:31, 11.63s/it][2022-12-20 23:37:47,428] [INFO] [timer.py:197:stop] 0/7892, RunningAvgSamplesPerSec=5.8788926945396, CurrSamplesPerSec=5.306178145276303, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3899/5000 [13:15:27<3:30:10, 11.45s/it][2022-12-20 23:37:58,458] [INFO] [timer.py:197:stop] 0/7894, RunningAvgSamplesPerSec=5.87888989066829, CurrSamplesPerSec=5.326595564024068, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3900/5000 [13:15:39<3:27:40, 11.33s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.3533333333333334e-06, 'epoch': 95.12}

 78%|███████▊  | 3900/5000 [13:15:39<3:27:40, 11.33s/it][2022-12-20 23:38:09,470] [INFO] [timer.py:197:stop] 0/7896, RunningAvgSamplesPerSec=5.878889686922154, CurrSamplesPerSec=5.344928889977026, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3901/5000 [13:15:50<3:25:52, 11.24s/it][2022-12-20 23:38:20,565] [INFO] [timer.py:197:stop] 0/7898, RunningAvgSamplesPerSec=5.878877871250663, CurrSamplesPerSec=5.290292385637862, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3902/5000 [13:16:01<3:24:46, 11.19s/it][2022-12-20 23:38:31,592] [INFO] [logging.py:68:log_dist] [Rank 0] step=3950, skipped=5, lr=[2.346666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:38:31,593] [INFO] [timer.py:197:stop] 0/7900, RunningAvgSamplesPerSec=5.878875086481545, CurrSamplesPerSec=5.328518238335531, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3903/5000 [13:16:12<3:23:36, 11.14s/it][2022-12-20 23:38:42,570] [INFO] [timer.py:197:stop] 0/7902, RunningAvgSamplesPerSec=5.8788794440153, CurrSamplesPerSec=5.361598139619385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3904/5000 [13:16:23<3:22:26, 11.08s/it][2022-12-20 23:38:53,556] [INFO] [timer.py:197:stop] 0/7904, RunningAvgSamplesPerSec=5.878882357836643, CurrSamplesPerSec=5.336097029395148, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3905/5000 [13:16:34<3:21:51, 11.06s/it][2022-12-20 23:39:04,563] [INFO] [timer.py:197:stop] 0/7906, RunningAvgSamplesPerSec=5.878882518073066, CurrSamplesPerSec=5.337262609178235, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3906/5000 [13:16:45<3:21:28, 11.05s/it][2022-12-20 23:39:15,609] [INFO] [timer.py:197:stop] 0/7908, RunningAvgSamplesPerSec=5.878877264090868, CurrSamplesPerSec=5.318523953457095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3907/5000 [13:16:56<3:21:13, 11.05s/it][2022-12-20 23:39:26,638] [INFO] [timer.py:197:stop] 0/7910, RunningAvgSamplesPerSec=5.878874390716469, CurrSamplesPerSec=5.326485219427959, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3908/5000 [13:17:07<3:20:53, 11.04s/it][2022-12-20 23:39:37,633] [INFO] [timer.py:197:stop] 0/7912, RunningAvgSamplesPerSec=5.878876239394385, CurrSamplesPerSec=5.348085405213583, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3909/5000 [13:17:18<3:20:21, 11.02s/it][2022-12-20 23:39:48,617] [INFO] [timer.py:197:stop] 0/7914, RunningAvgSamplesPerSec=5.878879531774835, CurrSamplesPerSec=5.339378419316736, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3910/5000 [13:17:29<3:19:59, 11.01s/it][2022-12-20 23:39:59,606] [INFO] [timer.py:197:stop] 0/7916, RunningAvgSamplesPerSec=5.878882352361135, CurrSamplesPerSec=5.335342740157891, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3911/5000 [13:17:40<3:19:52, 11.01s/it][2022-12-20 23:40:10,605] [INFO] [timer.py:197:stop] 0/7918, RunningAvgSamplesPerSec=5.878883693135093, CurrSamplesPerSec=5.355027453259119, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3912/5000 [13:17:51<3:19:19, 10.99s/it][2022-12-20 23:40:21,629] [INFO] [logging.py:68:log_dist] [Rank 0] step=3960, skipped=5, lr=[2.3244444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:40:21,631] [INFO] [timer.py:197:stop] 0/7920, RunningAvgSamplesPerSec=5.878881316859216, CurrSamplesPerSec=5.283400794735814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3913/5000 [13:18:02<3:19:24, 11.01s/it][2022-12-20 23:40:32,626] [INFO] [timer.py:197:stop] 0/7922, RunningAvgSamplesPerSec=5.878883307961018, CurrSamplesPerSec=5.324480365438023, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3914/5000 [13:18:13<3:19:22, 11.01s/it][2022-12-20 23:40:43,671] [INFO] [timer.py:197:stop] 0/7924, RunningAvgSamplesPerSec=5.87887845229876, CurrSamplesPerSec=5.314681976314301, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3915/5000 [13:18:24<3:19:07, 11.01s/it][2022-12-20 23:40:54,676] [INFO] [timer.py:197:stop] 0/7926, RunningAvgSamplesPerSec=5.87887896934134, CurrSamplesPerSec=5.312508096843493, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3916/5000 [13:18:35<3:19:02, 11.02s/it][2022-12-20 23:41:05,663] [INFO] [timer.py:197:stop] 0/7928, RunningAvgSamplesPerSec=5.8788820159449795, CurrSamplesPerSec=5.350831776999772, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3917/5000 [13:18:46<3:18:40, 11.01s/it][2022-12-20 23:41:16,646] [INFO] [timer.py:197:stop] 0/7930, RunningAvgSamplesPerSec=5.878885598039494, CurrSamplesPerSec=5.3520481939721645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3918/5000 [13:18:57<3:18:14, 10.99s/it][2022-12-20 23:41:27,665] [INFO] [timer.py:197:stop] 0/7932, RunningAvgSamplesPerSec=5.87888417912698, CurrSamplesPerSec=5.29993295009323, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3919/5000 [13:19:08<3:18:22, 11.01s/it][2022-12-20 23:41:38,664] [INFO] [timer.py:197:stop] 0/7934, RunningAvgSamplesPerSec=5.878885590039562, CurrSamplesPerSec=5.346700390249608, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3920/5000 [13:19:19<3:17:57, 11.00s/it][2022-12-20 23:41:49,669] [INFO] [timer.py:197:stop] 0/7936, RunningAvgSamplesPerSec=5.878886035599633, CurrSamplesPerSec=5.313889968308931, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3921/5000 [13:19:30<3:17:54, 11.01s/it][2022-12-20 23:42:00,637] [INFO] [timer.py:197:stop] 0/7938, RunningAvgSamplesPerSec=5.878891675688125, CurrSamplesPerSec=5.362221903940613, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3922/5000 [13:19:41<3:17:32, 10.99s/it][2022-12-20 23:42:11,653] [INFO] [logging.py:68:log_dist] [Rank 0] step=3970, skipped=5, lr=[2.302222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:42:11,655] [INFO] [timer.py:197:stop] 0/7940, RunningAvgSamplesPerSec=5.878890561891558, CurrSamplesPerSec=5.320429837120422, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3923/5000 [13:19:52<3:17:16, 10.99s/it][2022-12-20 23:42:22,644] [INFO] [timer.py:197:stop] 0/7942, RunningAvgSamplesPerSec=5.878893277883253, CurrSamplesPerSec=5.311251995908265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3924/5000 [13:20:03<3:17:12, 11.00s/it][2022-12-20 23:42:33,633] [INFO] [timer.py:197:stop] 0/7944, RunningAvgSamplesPerSec=5.878896089454, CurrSamplesPerSec=5.332834926138469, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 78%|███████▊  | 3925/5000 [13:20:14<3:17:05, 11.00s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.297777777777778e-06, 'epoch': 95.72}

 78%|███████▊  | 3925/5000 [13:20:14<3:17:05, 11.00s/it][2022-12-20 23:42:44,609] [INFO] [timer.py:197:stop] 0/7946, RunningAvgSamplesPerSec=5.878900982390727, CurrSamplesPerSec=5.3621908408139065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3926/5000 [13:20:25<3:16:39, 10.99s/it][2022-12-20 23:42:55,612] [INFO] [timer.py:197:stop] 0/7948, RunningAvgSamplesPerSec=5.878901890515244, CurrSamplesPerSec=5.318572216175446, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3927/5000 [13:20:36<3:16:33, 10.99s/it][2022-12-20 23:43:06,618] [INFO] [timer.py:197:stop] 0/7950, RunningAvgSamplesPerSec=5.878902450514792, CurrSamplesPerSec=5.3153559160710815, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3928/5000 [13:20:47<3:16:37, 11.00s/it][2022-12-20 23:43:17,615] [INFO] [timer.py:197:stop] 0/7952, RunningAvgSamplesPerSec=5.878903966784523, CurrSamplesPerSec=5.350079927688591, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3929/5000 [13:20:58<3:16:19, 11.00s/it][2022-12-20 23:43:28,583] [INFO] [timer.py:197:stop] 0/7954, RunningAvgSamplesPerSec=5.878909573692298, CurrSamplesPerSec=5.36369235127001, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3930/5000 [13:21:09<3:16:06, 11.00s/it][2022-12-20 23:43:39,629] [INFO] [timer.py:197:stop] 0/7956, RunningAvgSamplesPerSec=5.878904800100759, CurrSamplesPerSec=5.316927141217511, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3931/5000 [13:21:20<3:15:57, 11.00s/it][2022-12-20 23:43:50,621] [INFO] [timer.py:197:stop] 0/7958, RunningAvgSamplesPerSec=5.878907263834069, CurrSamplesPerSec=5.327062148125951, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3932/5000 [13:21:31<3:15:54, 11.01s/it][2022-12-20 23:44:01,650] [INFO] [logging.py:68:log_dist] [Rank 0] step=3980, skipped=5, lr=[2.28e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:44:01,651] [INFO] [timer.py:197:stop] 0/7960, RunningAvgSamplesPerSec=5.878904354090247, CurrSamplesPerSec=5.321340040930845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3933/5000 [13:21:42<3:15:43, 11.01s/it][2022-12-20 23:44:12,671] [INFO] [timer.py:197:stop] 0/7962, RunningAvgSamplesPerSec=5.878902854476758, CurrSamplesPerSec=5.306947291971816, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3934/5000 [13:21:53<3:15:49, 11.02s/it][2022-12-20 23:44:23,697] [INFO] [timer.py:197:stop] 0/7964, RunningAvgSamplesPerSec=5.878900636471695, CurrSamplesPerSec=5.3380885628621675, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3935/5000 [13:22:04<3:15:22, 11.01s/it][2022-12-20 23:44:34,690] [INFO] [timer.py:197:stop] 0/7966, RunningAvgSamplesPerSec=5.878902889348789, CurrSamplesPerSec=5.320294229697979, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3936/5000 [13:22:15<3:15:23, 11.02s/it][2022-12-20 23:44:44,818] [INFO] [timer.py:197:stop] 0/7968, RunningAvgSamplesPerSec=5.879021055678216, CurrSamplesPerSec=6.261743732470191, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-20 23:44:55,867] [INFO] [timer.py:197:stop] 0/7970, RunningAvgSamplesPerSec=5.8790154241969095, CurrSamplesPerSec=5.3110330011382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▊  | 3937/5000 [13:22:31<3:42:48, 12.58s/it][2022-12-20 23:45:06,832] [INFO] [timer.py:197:stop] 0/7972, RunningAvgSamplesPerSec=5.879021501316945, CurrSamplesPerSec=5.35462153910035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3938/5000 [13:22:42<3:34:02, 12.09s/it][2022-12-20 23:45:17,840] [INFO] [timer.py:197:stop] 0/7974, RunningAvgSamplesPerSec=5.879021763991139, CurrSamplesPerSec=5.315892327554655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3939/5000 [13:22:53<3:28:05, 11.77s/it][2022-12-20 23:45:28,880] [INFO] [timer.py:197:stop] 0/7976, RunningAvgSamplesPerSec=5.879017599329839, CurrSamplesPerSec=5.307782989861259, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3940/5000 [13:23:04<3:24:02, 11.55s/it][2022-12-20 23:45:39,882] [INFO] [timer.py:197:stop] 0/7978, RunningAvgSamplesPerSec=5.879018702932782, CurrSamplesPerSec=5.342815485319329, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3941/5000 [13:23:15<3:20:56, 11.39s/it][2022-12-20 23:45:50,870] [INFO] [logging.py:68:log_dist] [Rank 0] step=3990, skipped=5, lr=[2.257777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:45:50,871] [INFO] [timer.py:197:stop] 0/7980, RunningAvgSamplesPerSec=5.8790213438661, CurrSamplesPerSec=5.3508345501684715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3942/5000 [13:23:26<3:18:39, 11.27s/it][2022-12-20 23:46:01,861] [INFO] [timer.py:197:stop] 0/7982, RunningAvgSamplesPerSec=5.879023982058138, CurrSamplesPerSec=5.322068006172942, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3943/5000 [13:23:37<3:17:00, 11.18s/it][2022-12-20 23:46:12,850] [INFO] [timer.py:197:stop] 0/7984, RunningAvgSamplesPerSec=5.879026710812216, CurrSamplesPerSec=5.338748702929006, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3944/5000 [13:23:48<3:15:48, 11.13s/it][2022-12-20 23:46:23,882] [INFO] [timer.py:197:stop] 0/7986, RunningAvgSamplesPerSec=5.879023570951474, CurrSamplesPerSec=5.321751475074357, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3945/5000 [13:23:59<3:15:07, 11.10s/it][2022-12-20 23:46:34,905] [INFO] [timer.py:197:stop] 0/7988, RunningAvgSamplesPerSec=5.879021763185896, CurrSamplesPerSec=5.359743341338609, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3946/5000 [13:24:10<3:14:33, 11.07s/it][2022-12-20 23:46:45,936] [INFO] [timer.py:197:stop] 0/7990, RunningAvgSamplesPerSec=5.879018890392515, CurrSamplesPerSec=5.302349788212537, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3947/5000 [13:24:21<3:14:08, 11.06s/it][2022-12-20 23:46:57,006] [INFO] [timer.py:197:stop] 0/7992, RunningAvgSamplesPerSec=5.879010862722608, CurrSamplesPerSec=5.297363382387859, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3948/5000 [13:24:32<3:13:59, 11.06s/it][2022-12-20 23:47:08,048] [INFO] [timer.py:197:stop] 0/7994, RunningAvgSamplesPerSec=5.879006538187522, CurrSamplesPerSec=5.305335822532682, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3949/5000 [13:24:43<3:13:41, 11.06s/it][2022-12-20 23:47:19,009] [INFO] [timer.py:197:stop] 0/7996, RunningAvgSamplesPerSec=5.8790130042525535, CurrSamplesPerSec=5.314405672533149, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3950/5000 [13:24:54<3:13:00, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.24e-06, 'epoch': 96.34}

 79%|███████▉  | 3950/5000 [13:24:54<3:13:00, 11.03s/it][2022-12-20 23:47:30,063] [INFO] [timer.py:197:stop] 0/7998, RunningAvgSamplesPerSec=5.879007201965908, CurrSamplesPerSec=5.307726736739969, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3951/5000 [13:25:05<3:12:57, 11.04s/it][2022-12-20 23:47:41,111] [INFO] [logging.py:68:log_dist] [Rank 0] step=4000, skipped=5, lr=[2.235555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:47:41,112] [INFO] [timer.py:197:stop] 0/8000, RunningAvgSamplesPerSec=5.879001748581906, CurrSamplesPerSec=5.316080560381886, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3952/5000 [13:25:16<3:12:49, 11.04s/it][2022-12-20 23:47:52,189] [INFO] [timer.py:197:stop] 0/8002, RunningAvgSamplesPerSec=5.87899255494425, CurrSamplesPerSec=5.302552146299819, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3953/5000 [13:25:27<3:12:50, 11.05s/it][2022-12-20 23:48:03,145] [INFO] [timer.py:197:stop] 0/8004, RunningAvgSamplesPerSec=5.878999989229112, CurrSamplesPerSec=5.327957702715014, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3954/5000 [13:25:38<3:12:09, 11.02s/it][2022-12-20 23:48:14,153] [INFO] [timer.py:197:stop] 0/8006, RunningAvgSamplesPerSec=5.879000276817181, CurrSamplesPerSec=5.3453982657947305, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3955/5000 [13:25:49<3:11:53, 11.02s/it][2022-12-20 23:48:25,099] [INFO] [timer.py:197:stop] 0/8008, RunningAvgSamplesPerSec=5.879008876892938, CurrSamplesPerSec=5.357384894106234, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3956/5000 [13:26:00<3:11:20, 11.00s/it][2022-12-20 23:48:36,070] [INFO] [timer.py:197:stop] 0/8010, RunningAvgSamplesPerSec=5.87901425063256, CurrSamplesPerSec=5.33043256306842, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3957/5000 [13:26:11<3:11:01, 10.99s/it][2022-12-20 23:48:47,060] [INFO] [timer.py:197:stop] 0/8012, RunningAvgSamplesPerSec=5.879016874857364, CurrSamplesPerSec=5.370808461611475, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3958/5000 [13:26:22<3:10:50, 10.99s/it][2022-12-20 23:48:58,047] [INFO] [timer.py:197:stop] 0/8014, RunningAvgSamplesPerSec=5.879019720350586, CurrSamplesPerSec=5.34755099971453, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3959/5000 [13:26:33<3:10:39, 10.99s/it][2022-12-20 23:49:09,015] [INFO] [timer.py:197:stop] 0/8016, RunningAvgSamplesPerSec=5.87902509367868, CurrSamplesPerSec=5.33241605643001, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3960/5000 [13:26:44<3:10:21, 10.98s/it][2022-12-20 23:49:20,048] [INFO] [timer.py:197:stop] 0/8018, RunningAvgSamplesPerSec=5.87902175360841, CurrSamplesPerSec=5.321361560524614, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3961/5000 [13:26:55<3:10:26, 11.00s/it][2022-12-20 23:49:31,096] [INFO] [logging.py:68:log_dist] [Rank 0] step=4010, skipped=5, lr=[2.2133333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:49:31,098] [INFO] [timer.py:197:stop] 0/8020, RunningAvgSamplesPerSec=5.8790163010814975, CurrSamplesPerSec=5.310760228688641, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3962/5000 [13:27:06<3:10:31, 11.01s/it][2022-12-20 23:49:42,079] [INFO] [timer.py:197:stop] 0/8022, RunningAvgSamplesPerSec=5.879020066849477, CurrSamplesPerSec=5.355524247743994, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3963/5000 [13:27:17<3:10:10, 11.00s/it][2022-12-20 23:49:53,059] [INFO] [timer.py:197:stop] 0/8024, RunningAvgSamplesPerSec=5.879023871320158, CurrSamplesPerSec=5.364921485431285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3964/5000 [13:27:28<3:09:52, 11.00s/it][2022-12-20 23:50:03,994] [INFO] [timer.py:197:stop] 0/8026, RunningAvgSamplesPerSec=5.879033777735862, CurrSamplesPerSec=5.361430442031898, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3965/5000 [13:27:39<3:09:22, 10.98s/it][2022-12-20 23:50:14,980] [INFO] [timer.py:197:stop] 0/8028, RunningAvgSamplesPerSec=5.879036850889962, CurrSamplesPerSec=5.330978585960009, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3966/5000 [13:27:50<3:09:13, 10.98s/it][2022-12-20 23:50:25,986] [INFO] [timer.py:197:stop] 0/8030, RunningAvgSamplesPerSec=5.879037041637896, CurrSamplesPerSec=5.309151693401624, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3967/5000 [13:28:01<3:09:10, 10.99s/it][2022-12-20 23:50:37,031] [INFO] [timer.py:197:stop] 0/8032, RunningAvgSamplesPerSec=5.879032263377862, CurrSamplesPerSec=5.296480172451452, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3968/5000 [13:28:12<3:09:17, 11.01s/it][2022-12-20 23:50:48,013] [INFO] [timer.py:197:stop] 0/8034, RunningAvgSamplesPerSec=5.879035718639782, CurrSamplesPerSec=5.358451971236659, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3969/5000 [13:28:23<3:08:59, 11.00s/it][2022-12-20 23:50:59,143] [INFO] [timer.py:197:stop] 0/8036, RunningAvgSamplesPerSec=5.879019486700361, CurrSamplesPerSec=5.2303435203746025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3970/5000 [13:28:34<3:09:29, 11.04s/it][2022-12-20 23:51:10,153] [INFO] [timer.py:197:stop] 0/8038, RunningAvgSamplesPerSec=5.879019682155673, CurrSamplesPerSec=5.344526633451091, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3971/5000 [13:28:45<3:09:09, 11.03s/it][2022-12-20 23:51:21,146] [INFO] [logging.py:68:log_dist] [Rank 0] step=4020, skipped=5, lr=[2.1911111111111115e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:51:21,148] [INFO] [timer.py:197:stop] 0/8040, RunningAvgSamplesPerSec=5.879021766128096, CurrSamplesPerSec=5.323278768471806, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3972/5000 [13:28:56<3:08:47, 11.02s/it][2022-12-20 23:51:32,141] [INFO] [timer.py:197:stop] 0/8042, RunningAvgSamplesPerSec=5.87902404473087, CurrSamplesPerSec=5.36313853526619, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3973/5000 [13:29:07<3:08:28, 11.01s/it][2022-12-20 23:51:43,150] [INFO] [timer.py:197:stop] 0/8044, RunningAvgSamplesPerSec=5.87902416724383, CurrSamplesPerSec=5.308165878601749, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 79%|███████▉  | 3974/5000 [13:29:18<3:08:16, 11.01s/it][2022-12-20 23:51:54,107] [INFO] [timer.py:197:stop] 0/8046, RunningAvgSamplesPerSec=5.879031237520502, CurrSamplesPerSec=5.357375698856583, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3975/5000 [13:29:29<3:07:49, 10.99s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.1844444444444446e-06, 'epoch': 96.94}

 80%|███████▉  | 3975/5000 [13:29:29<3:07:49, 10.99s/it][2022-12-20 23:52:05,122] [INFO] [timer.py:197:stop] 0/8048, RunningAvgSamplesPerSec=5.87903079854065, CurrSamplesPerSec=5.325618898418627, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3976/5000 [13:29:40<3:07:44, 11.00s/it][2022-12-20 23:52:16,091] [INFO] [timer.py:197:stop] 0/8050, RunningAvgSamplesPerSec=5.87903638791964, CurrSamplesPerSec=5.326353741989549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3977/5000 [13:29:51<3:07:23, 10.99s/it][2022-12-20 23:52:26,190] [INFO] [timer.py:197:stop] 0/8052, RunningAvgSamplesPerSec=5.879157287656026, CurrSamplesPerSec=5.311837191452507, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3978/5000 [13:30:06<3:27:44, 12.20s/it][2022-12-20 23:52:37,195] [INFO] [timer.py:197:stop] 0/8054, RunningAvgSamplesPerSec=5.879157810484929, CurrSamplesPerSec=5.304329413047954, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3979/5000 [13:30:17<3:21:32, 11.84s/it][2022-12-20 23:52:48,211] [INFO] [timer.py:197:stop] 0/8056, RunningAvgSamplesPerSec=5.879156832409986, CurrSamplesPerSec=5.3089267820073145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3980/5000 [13:30:28<3:17:25, 11.61s/it][2022-12-20 23:52:59,244] [INFO] [timer.py:197:stop] 0/8058, RunningAvgSamplesPerSec=5.879153641081622, CurrSamplesPerSec=5.347164964584492, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3981/5000 [13:30:39<3:13:52, 11.42s/it][2022-12-20 23:53:10,244] [INFO] [logging.py:68:log_dist] [Rank 0] step=4030, skipped=5, lr=[2.168888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:53:10,245] [INFO] [timer.py:197:stop] 0/8060, RunningAvgSamplesPerSec=5.879154740709566, CurrSamplesPerSec=5.3066789255054525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3982/5000 [13:30:50<3:11:41, 11.30s/it][2022-12-20 23:53:21,233] [INFO] [timer.py:197:stop] 0/8062, RunningAvgSamplesPerSec=5.879157468337238, CurrSamplesPerSec=5.337938466948506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3983/5000 [13:31:01<3:10:05, 11.21s/it][2022-12-20 23:53:32,245] [INFO] [timer.py:197:stop] 0/8064, RunningAvgSamplesPerSec=5.879157050450306, CurrSamplesPerSec=5.345096408302825, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3984/5000 [13:31:12<3:08:42, 11.14s/it][2022-12-20 23:53:43,254] [INFO] [timer.py:197:stop] 0/8066, RunningAvgSamplesPerSec=5.87915736363983, CurrSamplesPerSec=5.319964835792752, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3985/5000 [13:31:23<3:07:44, 11.10s/it][2022-12-20 23:53:54,283] [INFO] [timer.py:197:stop] 0/8068, RunningAvgSamplesPerSec=5.879154752741992, CurrSamplesPerSec=5.286399249320175, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3986/5000 [13:31:34<3:07:18, 11.08s/it][2022-12-20 23:54:05,325] [INFO] [timer.py:197:stop] 0/8070, RunningAvgSamplesPerSec=5.879150298300576, CurrSamplesPerSec=5.291553820393454, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3987/5000 [13:31:45<3:07:04, 11.08s/it][2022-12-20 23:54:16,352] [INFO] [timer.py:197:stop] 0/8072, RunningAvgSamplesPerSec=5.879147922641214, CurrSamplesPerSec=5.333284590424864, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3988/5000 [13:31:56<3:06:18, 11.05s/it][2022-12-20 23:54:27,328] [INFO] [timer.py:197:stop] 0/8074, RunningAvgSamplesPerSec=5.879152456155177, CurrSamplesPerSec=5.32535772591076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3989/5000 [13:32:07<3:06:02, 11.04s/it][2022-12-20 23:54:38,389] [INFO] [timer.py:197:stop] 0/8076, RunningAvgSamplesPerSec=5.879145528684287, CurrSamplesPerSec=5.296573601086199, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3990/5000 [13:32:18<3:05:58, 11.05s/it][2022-12-20 23:54:49,448] [INFO] [timer.py:197:stop] 0/8078, RunningAvgSamplesPerSec=5.879138868083944, CurrSamplesPerSec=5.302382047162607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3991/5000 [13:32:29<3:05:48, 11.05s/it][2022-12-20 23:55:00,468] [INFO] [logging.py:68:log_dist] [Rank 0] step=4040, skipped=5, lr=[2.1466666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-20 23:55:00,470] [INFO] [timer.py:197:stop] 0/8080, RunningAvgSamplesPerSec=5.879137156472995, CurrSamplesPerSec=5.328352392116208, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3992/5000 [13:32:41<3:05:28, 11.04s/it][2022-12-20 23:55:11,463] [INFO] [timer.py:197:stop] 0/8082, RunningAvgSamplesPerSec=5.879139299929784, CurrSamplesPerSec=5.350232200101593, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3993/5000 [13:32:52<3:05:09, 11.03s/it][2022-12-20 23:55:22,501] [INFO] [timer.py:197:stop] 0/8084, RunningAvgSamplesPerSec=5.879135299194591, CurrSamplesPerSec=5.328872599991503, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3994/5000 [13:33:03<3:04:58, 11.03s/it][2022-12-20 23:55:33,537] [INFO] [timer.py:197:stop] 0/8086, RunningAvgSamplesPerSec=5.879131861713902, CurrSamplesPerSec=5.327379099848214, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3995/5000 [13:33:14<3:04:36, 11.02s/it][2022-12-20 23:55:44,542] [INFO] [timer.py:197:stop] 0/8088, RunningAvgSamplesPerSec=5.879132454490535, CurrSamplesPerSec=5.320064155780077, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3996/5000 [13:33:25<3:04:22, 11.02s/it][2022-12-20 23:55:55,552] [INFO] [timer.py:197:stop] 0/8090, RunningAvgSamplesPerSec=5.87913239966328, CurrSamplesPerSec=5.319070490369771, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3997/5000 [13:33:36<3:04:20, 11.03s/it][2022-12-20 23:56:06,581] [INFO] [timer.py:197:stop] 0/8092, RunningAvgSamplesPerSec=5.87912963128783, CurrSamplesPerSec=5.3362958188408625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3998/5000 [13:33:47<3:04:01, 11.02s/it][2022-12-20 23:56:17,557] [INFO] [timer.py:197:stop] 0/8094, RunningAvgSamplesPerSec=5.879133926343883, CurrSamplesPerSec=5.359716587509476, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|███████▉  | 3999/5000 [13:33:58<3:03:28, 11.00s/it][2022-12-20 23:56:28,506] [INFO] [timer.py:197:stop] 0/8096, RunningAvgSamplesPerSec=5.879141902984215, CurrSamplesPerSec=5.35595829720102, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4000/5000 [13:34:09<3:03:08, 10.99s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.128888888888889e-06, 'epoch': 97.55}

 80%|████████  | 4000/5000 [13:34:09<3:03:08, 10.99s/it][INFO|trainer.py:2955] 2022-12-20 23:56:33,442 >> ***** Running Evaluation *****
[INFO|trainer.py:2957] 2022-12-20 23:56:33,442 >>   Num examples = 1325
[INFO|trainer.py:2960] 2022-12-20 23:56:33,442 >>   Batch size = 8


  0%|          | 0/166 [00:00<?, ?it/s]

  1%|          | 2/166 [00:05<07:53,  2.88s/it]

  2%|▏         | 3/166 [00:11<11:09,  4.11s/it]

  2%|▏         | 4/166 [00:17<12:55,  4.79s/it]

  3%|▎         | 5/166 [00:24<15:06,  5.63s/it]

  4%|▎         | 6/166 [00:32<16:30,  6.19s/it]

  4%|▍         | 7/166 [00:38<16:30,  6.23s/it]

  5%|▍         | 8/166 [00:44<16:06,  6.12s/it]

  5%|▌         | 9/166 [00:50<15:51,  6.06s/it]

  6%|▌         | 10/166 [00:55<15:31,  5.97s/it]

  7%|▋         | 11/166 [01:01<15:08,  5.86s/it]

  7%|▋         | 12/166 [01:07<15:08,  5.90s/it]

  8%|▊         | 13/166 [01:14<15:29,  6.07s/it]

  8%|▊         | 14/166 [01:20<16:02,  6.33s/it]

  9%|▉         | 15/166 [01:26<15:25,  6.13s/it]

 10%|▉         | 16/166 [01:33<16:13,  6.49s/it]

 10%|█         | 17/166 [01:40<16:16,  6.55s/it]

 11%|█         | 18/166 [01:46<15:56,  6.46s/it]

 11%|█▏        | 19/166 [01:52<15:24,  6.29s/it]

 12%|█▏        | 20/166 [02:03<18:32,  7.62s/it]

 13%|█▎        | 21/166 [02:10<18:07,  7.50s/it]

 13%|█▎        | 22/166 [02:18<17:52,  7.45s/it]

 14%|█▍        | 23/166 [02:25<17:50,  7.48s/it]

 14%|█▍        | 24/166 [02:32<16:59,  7.18s/it]

 15%|█▌        | 25/166 [02:39<16:53,  7.19s/it]

 16%|█▌        | 26/166 [02:46<16:40,  7.15s/it]

 16%|█▋        | 27/166 [02:53<16:43,  7.22s/it]

 17%|█▋        | 28/166 [02:59<15:46,  6.86s/it]

 17%|█▋        | 29/166 [03:07<16:35,  7.27s/it]

 18%|█▊        | 30/166 [03:14<16:11,  7.15s/it]

 19%|█▊        | 31/166 [03:22<16:09,  7.18s/it]

 19%|█▉        | 32/166 [03:29<16:22,  7.34s/it]

 20%|█▉        | 33/166 [03:37<16:18,  7.36s/it]

 20%|██        | 34/166 [03:43<15:34,  7.08s/it]

 21%|██        | 35/166 [03:51<16:01,  7.34s/it]

 22%|██▏       | 36/166 [03:58<15:57,  7.37s/it]

 22%|██▏       | 37/166 [04:05<15:18,  7.12s/it]

 23%|██▎       | 38/166 [04:14<16:05,  7.54s/it]

 23%|██▎       | 39/166 [04:20<15:32,  7.34s/it]

 24%|██▍       | 40/166 [04:28<15:40,  7.47s/it]

 25%|██▍       | 41/166 [04:35<15:04,  7.24s/it]

 25%|██▌       | 42/166 [04:42<14:37,  7.08s/it]

 26%|██▌       | 43/166 [04:48<14:06,  6.88s/it]

 27%|██▋       | 44/166 [04:57<15:00,  7.38s/it]

 27%|██▋       | 45/166 [05:05<15:18,  7.59s/it]

 28%|██▊       | 46/166 [05:15<16:41,  8.34s/it]

 28%|██▊       | 47/166 [05:23<16:28,  8.30s/it]

 29%|██▉       | 48/166 [05:30<15:37,  7.94s/it]

 30%|██▉       | 49/166 [05:36<14:16,  7.32s/it]

 30%|███       | 50/166 [05:44<14:51,  7.69s/it]

 31%|███       | 51/166 [05:52<14:35,  7.61s/it]

 31%|███▏      | 52/166 [06:00<14:28,  7.62s/it]

 32%|███▏      | 53/166 [06:07<14:05,  7.48s/it]

 33%|███▎      | 54/166 [06:15<14:13,  7.62s/it]

 33%|███▎      | 55/166 [06:21<13:09,  7.12s/it]

 34%|███▎      | 56/166 [06:27<12:32,  6.84s/it]

 34%|███▍      | 57/166 [06:35<13:08,  7.24s/it]

 35%|███▍      | 58/166 [06:41<12:07,  6.73s/it]

 36%|███▌      | 59/166 [06:48<12:12,  6.84s/it]

 36%|███▌      | 60/166 [06:55<12:15,  6.94s/it]

 37%|███▋      | 61/166 [07:04<13:17,  7.60s/it]

 37%|███▋      | 62/166 [07:11<13:01,  7.52s/it]

 38%|███▊      | 63/166 [07:19<12:46,  7.44s/it]

 39%|███▊      | 64/166 [07:28<13:28,  7.93s/it]

 39%|███▉      | 65/166 [07:36<13:37,  8.09s/it]

 40%|███▉      | 66/166 [07:43<12:52,  7.73s/it]

 40%|████      | 67/166 [07:50<12:21,  7.49s/it]

 41%|████      | 68/166 [07:55<11:04,  6.79s/it]

 42%|████▏     | 69/166 [08:02<11:00,  6.81s/it]

 42%|████▏     | 70/166 [08:10<11:28,  7.18s/it]

 43%|████▎     | 71/166 [08:18<11:42,  7.39s/it]

 43%|████▎     | 72/166 [08:24<11:09,  7.12s/it]

 44%|████▍     | 73/166 [08:31<10:42,  6.91s/it]

 45%|████▍     | 74/166 [08:37<10:19,  6.73s/it]

 45%|████▌     | 75/166 [08:43<10:01,  6.61s/it]

 46%|████▌     | 76/166 [08:51<10:14,  6.82s/it]

 46%|████▋     | 77/166 [08:56<09:39,  6.51s/it]

 47%|████▋     | 78/166 [09:04<09:48,  6.69s/it]

 48%|████▊     | 79/166 [09:12<10:15,  7.07s/it]

 48%|████▊     | 80/166 [09:20<10:36,  7.40s/it]

 49%|████▉     | 81/166 [09:28<10:41,  7.55s/it]

 49%|████▉     | 82/166 [09:35<10:18,  7.36s/it]

 50%|█████     | 83/166 [09:40<09:20,  6.76s/it]

 51%|█████     | 84/166 [09:46<09:06,  6.66s/it]

 51%|█████     | 85/166 [09:54<09:15,  6.86s/it]

 52%|█████▏    | 86/166 [10:02<09:43,  7.29s/it]

 52%|█████▏    | 87/166 [10:09<09:42,  7.38s/it]

 53%|█████▎    | 88/166 [10:16<09:19,  7.18s/it]

 54%|█████▎    | 89/166 [10:24<09:16,  7.23s/it]

 54%|█████▍    | 90/166 [10:30<08:53,  7.02s/it]

 55%|█████▍    | 91/166 [10:37<08:39,  6.93s/it]

 55%|█████▌    | 92/166 [10:43<08:12,  6.66s/it]

 56%|█████▌    | 93/166 [10:51<08:37,  7.09s/it]

 57%|█████▋    | 94/166 [11:01<09:36,  8.01s/it]

 57%|█████▋    | 95/166 [11:06<08:30,  7.18s/it]

 58%|█████▊    | 96/166 [11:13<08:09,  6.99s/it]

 58%|█████▊    | 97/166 [11:23<09:09,  7.97s/it]

 59%|█████▉    | 98/166 [11:33<09:41,  8.55s/it]

 60%|█████▉    | 99/166 [11:40<09:10,  8.22s/it]

 60%|██████    | 100/166 [11:47<08:22,  7.61s/it]

 61%|██████    | 101/166 [11:53<07:55,  7.31s/it]

 61%|██████▏   | 102/166 [12:02<08:17,  7.78s/it]

 62%|██████▏   | 103/166 [12:11<08:31,  8.12s/it]

 63%|██████▎   | 104/166 [12:18<08:07,  7.87s/it]

 63%|██████▎   | 105/166 [12:27<08:06,  7.98s/it]

 64%|██████▍   | 106/166 [12:36<08:31,  8.53s/it]

 64%|██████▍   | 107/166 [12:46<08:41,  8.83s/it]

 65%|██████▌   | 108/166 [12:51<07:33,  7.82s/it]

 66%|██████▌   | 109/166 [12:58<07:01,  7.39s/it]

 66%|██████▋   | 110/166 [13:05<06:54,  7.41s/it]

 67%|██████▋   | 111/166 [13:11<06:23,  6.98s/it]

 67%|██████▋   | 112/166 [13:19<06:22,  7.09s/it]

 68%|██████▊   | 113/166 [13:27<06:38,  7.53s/it]

 69%|██████▊   | 114/166 [13:35<06:30,  7.50s/it]

 69%|██████▉   | 115/166 [13:41<06:04,  7.15s/it]

 70%|██████▉   | 116/166 [13:47<05:44,  6.88s/it]

 70%|███████   | 117/166 [13:55<05:46,  7.07s/it]

 71%|███████   | 118/166 [14:02<05:50,  7.30s/it]

 72%|███████▏  | 119/166 [14:10<05:48,  7.42s/it]

 72%|███████▏  | 120/166 [14:18<05:52,  7.66s/it]

 73%|███████▎  | 121/166 [14:27<05:53,  7.86s/it]

 73%|███████▎  | 122/166 [14:34<05:43,  7.80s/it]

 74%|███████▍  | 123/166 [14:42<05:29,  7.65s/it]

 75%|███████▍  | 124/166 [14:48<05:10,  7.39s/it]

 75%|███████▌  | 125/166 [14:56<05:03,  7.40s/it]

 76%|███████▌  | 126/166 [15:04<05:08,  7.72s/it]

 77%|███████▋  | 127/166 [15:09<04:29,  6.90s/it]

 77%|███████▋  | 128/166 [15:19<04:58,  7.86s/it]

 78%|███████▊  | 129/166 [15:27<04:51,  7.87s/it]

 78%|███████▊  | 130/166 [15:34<04:32,  7.57s/it]

 79%|███████▉  | 131/166 [15:41<04:19,  7.41s/it]

 80%|███████▉  | 132/166 [15:49<04:18,  7.61s/it]

 80%|████████  | 133/166 [15:57<04:12,  7.66s/it]

 81%|████████  | 134/166 [16:04<03:54,  7.34s/it]

 81%|████████▏ | 135/166 [16:10<03:36,  6.98s/it]

 82%|████████▏ | 136/166 [16:20<04:00,  8.02s/it]

 83%|████████▎ | 137/166 [16:29<04:01,  8.31s/it]

 83%|████████▎ | 138/166 [16:39<04:07,  8.85s/it]

 84%|████████▎ | 139/166 [16:48<03:54,  8.68s/it]

 84%|████████▍ | 140/166 [16:54<03:27,  7.97s/it]

 85%|████████▍ | 141/166 [17:00<03:04,  7.36s/it]

 86%|████████▌ | 142/166 [17:07<02:55,  7.30s/it]

 86%|████████▌ | 143/166 [17:13<02:35,  6.76s/it]

 87%|████████▋ | 144/166 [17:21<02:42,  7.37s/it]

 87%|████████▋ | 145/166 [17:31<02:50,  8.10s/it]

 88%|████████▊ | 146/166 [17:41<02:49,  8.47s/it]

 89%|████████▊ | 147/166 [17:49<02:40,  8.47s/it]

 89%|████████▉ | 148/166 [17:57<02:32,  8.45s/it]

 90%|████████▉ | 149/166 [18:07<02:27,  8.67s/it]

 90%|█████████ | 150/166 [18:16<02:23,  8.97s/it]

 91%|█████████ | 151/166 [18:24<02:10,  8.68s/it]

 92%|█████████▏| 152/166 [18:30<01:49,  7.81s/it]

 92%|█████████▏| 153/166 [18:36<01:36,  7.41s/it]

 93%|█████████▎| 154/166 [18:44<01:28,  7.38s/it]

 93%|█████████▎| 155/166 [18:52<01:22,  7.50s/it]

 94%|█████████▍| 156/166 [18:57<01:10,  7.01s/it]

 95%|█████████▍| 157/166 [19:04<01:02,  6.92s/it]

 95%|█████████▌| 158/166 [19:10<00:52,  6.52s/it]

 96%|█████████▌| 159/166 [19:15<00:41,  5.99s/it]

 96%|█████████▋| 160/166 [19:20<00:34,  5.71s/it]

 97%|█████████▋| 161/166 [19:26<00:29,  5.82s/it]

 98%|█████████▊| 162/166 [19:33<00:24,  6.15s/it]

 98%|█████████▊| 163/166 [19:40<00:19,  6.45s/it]

 99%|█████████▉| 164/166 [19:46<00:12,  6.36s/it]

 99%|█████████▉| 165/166 [19:52<00:06,  6.23s/it]

100%|██████████| 166/166 [19:56<00:00,  5.63s/it]
                                                        

                                                 
{'eval_loss': 0.287109375, 'eval_wer': 8.482348391374007, 'eval_runtime': 1274.1311, 'eval_samples_per_second': 1.04, 'eval_steps_per_second': 0.13, 'epoch': 97.55}

 80%|████████  | 4000/5000 [13:55:23<3:03:08, 10.99s/it]

100%|██████████| 166/166 [21:08<00:00,  5.63s/it]

                                                 [INFO|trainer.py:2700] 2022-12-21 00:17:47,575 >> Saving model checkpoint to ./checkpoint-4000
[INFO|configuration_utils.py:447] 2022-12-21 00:17:47,576 >> Configuration saved in ./checkpoint-4000/config.json
[INFO|modeling_utils.py:1680] 2022-12-21 00:17:51,119 >> Model weights saved in ./checkpoint-4000/pytorch_model.bin
[INFO|feature_extraction_utils.py:368] 2022-12-21 00:17:51,134 >> Feature extractor saved in ./checkpoint-4000/preprocessor_config.json
[2022-12-21 00:17:51,134] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step4048 is begin to save!
/home/milan/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
[2022-12-21 00:17:51,144] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-4000/global_step4048/mp_rank_00_model_states.pt
[2022-12-21 00:17:51,144] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-4000/global_step4048/mp_rank_00_model_states.pt...
[2022-12-21 00:17:54,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-4000/global_step4048/mp_rank_00_model_states.pt.
[2022-12-21 00:17:54,993] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-4000/global_step4048/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2022-12-21 00:18:12,523] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-4000/global_step4048/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2022-12-21 00:18:12,524] [INFO] [engine.py:3269:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-4000/global_step4048/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2022-12-21 00:18:12,524] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4048 is ready now!
[INFO|feature_extraction_utils.py:368] 2022-12-21 00:18:17,037 >> Feature extractor saved in ./preprocessor_config.json
[2022-12-21 00:20:25,579] [INFO] [timer.py:197:stop] 0/8098, RunningAvgSamplesPerSec=5.879106476468289, CurrSamplesPerSec=5.093867850109623, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4001/5000 [13:58:06<121:46:12, 438.81s/it][2022-12-21 00:20:36,552] [INFO] [logging.py:68:log_dist] [Rank 0] step=4050, skipped=5, lr=[2.1244444444444443e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:20:36,553] [INFO] [timer.py:197:stop] 0/8100, RunningAvgSamplesPerSec=5.879111191728076, CurrSamplesPerSec=5.344233386168689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4002/5000 [13:58:17<86:03:58, 310.46s/it] [2022-12-21 00:20:47,528] [INFO] [timer.py:197:stop] 0/8102, RunningAvgSamplesPerSec=5.879115681926362, CurrSamplesPerSec=5.340438972256464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4003/5000 [13:58:28<61:06:00, 220.62s/it][2022-12-21 00:20:58,499] [INFO] [timer.py:197:stop] 0/8104, RunningAvgSamplesPerSec=5.879120604860978, CurrSamplesPerSec=5.368241653516812, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4004/5000 [13:58:38<43:37:59, 157.71s/it][2022-12-21 00:21:09,452] [INFO] [timer.py:197:stop] 0/8106, RunningAvgSamplesPerSec=5.879128004984679, CurrSamplesPerSec=5.334036813902272, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4005/5000 [13:58:49<31:25:16, 113.68s/it][2022-12-21 00:21:20,438] [INFO] [timer.py:197:stop] 0/8108, RunningAvgSamplesPerSec=5.879131214988293, CurrSamplesPerSec=5.310336205125673, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4006/5000 [13:59:00<22:53:14, 82.89s/it] [2022-12-21 00:21:31,473] [INFO] [timer.py:197:stop] 0/8110, RunningAvgSamplesPerSec=5.879127594218704, CurrSamplesPerSec=5.315880958213466, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4007/5000 [13:59:12<16:55:04, 61.33s/it][2022-12-21 00:21:42,451] [INFO] [timer.py:197:stop] 0/8112, RunningAvgSamplesPerSec=5.879131823148013, CurrSamplesPerSec=5.365864567875278, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4008/5000 [13:59:22<12:44:07, 46.22s/it][2022-12-21 00:21:53,404] [INFO] [timer.py:197:stop] 0/8114, RunningAvgSamplesPerSec=5.879139519458495, CurrSamplesPerSec=5.358554872858129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4009/5000 [13:59:33<9:48:47, 35.65s/it] [2022-12-21 00:22:04,426] [INFO] [timer.py:197:stop] 0/8116, RunningAvgSamplesPerSec=5.879137829975435, CurrSamplesPerSec=5.326190144079811, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4010/5000 [13:59:44<7:46:14, 28.26s/it][2022-12-21 00:22:15,421] [INFO] [timer.py:197:stop] 0/8118, RunningAvgSamplesPerSec=5.87913966067019, CurrSamplesPerSec=5.339037950493925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4011/5000 [13:59:55<6:20:19, 23.07s/it][2022-12-21 00:22:26,384] [INFO] [logging.py:68:log_dist] [Rank 0] step=4060, skipped=5, lr=[2.1022222222222224e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:22:26,385] [INFO] [timer.py:197:stop] 0/8120, RunningAvgSamplesPerSec=5.8791455071835035, CurrSamplesPerSec=5.351907982309296, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4012/5000 [14:00:06<5:20:15, 19.45s/it][2022-12-21 00:22:37,424] [INFO] [timer.py:197:stop] 0/8122, RunningAvgSamplesPerSec=5.879141627214186, CurrSamplesPerSec=5.309427661146448, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4013/5000 [14:00:17<4:38:26, 16.93s/it][2022-12-21 00:22:48,408] [INFO] [timer.py:197:stop] 0/8124, RunningAvgSamplesPerSec=5.879145044906297, CurrSamplesPerSec=5.361578649322693, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4014/5000 [14:00:28<4:08:47, 15.14s/it][2022-12-21 00:22:59,393] [INFO] [timer.py:197:stop] 0/8126, RunningAvgSamplesPerSec=5.879148138633306, CurrSamplesPerSec=5.346817538017743, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4015/5000 [14:00:39<3:47:59, 13.89s/it][2022-12-21 00:23:10,359] [INFO] [timer.py:197:stop] 0/8128, RunningAvgSamplesPerSec=5.879154032939573, CurrSamplesPerSec=5.350251181444112, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4016/5000 [14:00:50<3:33:33, 13.02s/it][2022-12-21 00:23:21,399] [INFO] [timer.py:197:stop] 0/8130, RunningAvgSamplesPerSec=5.8791499275397605, CurrSamplesPerSec=5.315678634182523, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4017/5000 [14:01:01<3:23:25, 12.42s/it][2022-12-21 00:23:32,345] [INFO] [timer.py:197:stop] 0/8132, RunningAvgSamplesPerSec=5.879158380112219, CurrSamplesPerSec=5.367233345519316, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4018/5000 [14:01:12<3:16:10, 11.99s/it][2022-12-21 00:23:42,466] [INFO] [timer.py:197:stop] 0/8134, RunningAvgSamplesPerSec=5.879275089343199, CurrSamplesPerSec=6.260939885516436, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 00:23:53,499] [INFO] [timer.py:197:stop] 0/8136, RunningAvgSamplesPerSec=5.879271499843242, CurrSamplesPerSec=5.310662096302032, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4019/5000 [14:01:29<3:36:38, 13.25s/it][2022-12-21 00:24:04,489] [INFO] [timer.py:197:stop] 0/8138, RunningAvgSamplesPerSec=5.8792741510805, CurrSamplesPerSec=5.306878886370058, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4020/5000 [14:01:40<3:25:20, 12.57s/it][2022-12-21 00:24:15,441] [INFO] [logging.py:68:log_dist] [Rank 0] step=4070, skipped=5, lr=[2.08e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:24:15,442] [INFO] [timer.py:197:stop] 0/8140, RunningAvgSamplesPerSec=5.879281779396381, CurrSamplesPerSec=5.359638039810382, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4021/5000 [14:01:51<3:17:12, 12.09s/it][2022-12-21 00:24:26,443] [INFO] [timer.py:197:stop] 0/8142, RunningAvgSamplesPerSec=5.879282888934946, CurrSamplesPerSec=5.350252247815665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4022/5000 [14:02:02<3:11:41, 11.76s/it][2022-12-21 00:24:37,436] [INFO] [timer.py:197:stop] 0/8144, RunningAvgSamplesPerSec=5.879285093361157, CurrSamplesPerSec=5.3417701458130535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4023/5000 [14:02:13<3:07:45, 11.53s/it][2022-12-21 00:24:48,414] [INFO] [timer.py:197:stop] 0/8146, RunningAvgSamplesPerSec=5.879289455792325, CurrSamplesPerSec=5.340900121248683, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4024/5000 [14:02:24<3:04:51, 11.36s/it][2022-12-21 00:24:59,386] [INFO] [timer.py:197:stop] 0/8148, RunningAvgSamplesPerSec=5.87929447814198, CurrSamplesPerSec=5.3434887082552285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 80%|████████  | 4025/5000 [14:02:34<3:02:45, 11.25s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.0711111111111114e-06, 'epoch': 98.17}

 80%|████████  | 4025/5000 [14:02:34<3:02:45, 11.25s/it][2022-12-21 00:25:10,344] [INFO] [timer.py:197:stop] 0/8150, RunningAvgSamplesPerSec=5.879301661537801, CurrSamplesPerSec=5.3441580578820895, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4026/5000 [14:02:45<3:01:09, 11.16s/it][2022-12-21 00:25:21,282] [INFO] [timer.py:197:stop] 0/8152, RunningAvgSamplesPerSec=5.879311095349123, CurrSamplesPerSec=5.331857665831148, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4027/5000 [14:02:56<2:59:53, 11.09s/it][2022-12-21 00:25:32,309] [INFO] [timer.py:197:stop] 0/8154, RunningAvgSamplesPerSec=5.879308716138182, CurrSamplesPerSec=5.311412155148097, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4028/5000 [14:03:07<2:59:23, 11.07s/it][2022-12-21 00:25:43,295] [INFO] [timer.py:197:stop] 0/8156, RunningAvgSamplesPerSec=5.879311798430926, CurrSamplesPerSec=5.346016562735164, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4029/5000 [14:03:18<2:58:47, 11.05s/it][2022-12-21 00:25:54,241] [INFO] [timer.py:197:stop] 0/8158, RunningAvgSamplesPerSec=5.879320160512912, CurrSamplesPerSec=5.369269886546951, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4030/5000 [14:03:29<2:58:06, 11.02s/it][2022-12-21 00:26:05,221] [INFO] [logging.py:68:log_dist] [Rank 0] step=4080, skipped=5, lr=[2.057777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:26:05,222] [INFO] [timer.py:197:stop] 0/8160, RunningAvgSamplesPerSec=5.879323786823011, CurrSamplesPerSec=5.358107354464163, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4031/5000 [14:03:40<2:57:44, 11.01s/it][2022-12-21 00:26:16,182] [INFO] [timer.py:197:stop] 0/8162, RunningAvgSamplesPerSec=5.8793302130287195, CurrSamplesPerSec=5.320550265681192, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4032/5000 [14:03:51<2:57:20, 10.99s/it][2022-12-21 00:26:27,180] [INFO] [timer.py:197:stop] 0/8164, RunningAvgSamplesPerSec=5.8793317793904345, CurrSamplesPerSec=5.3537840549629365, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4033/5000 [14:04:02<2:57:11, 10.99s/it][2022-12-21 00:26:38,202] [INFO] [timer.py:197:stop] 0/8166, RunningAvgSamplesPerSec=5.879330078091181, CurrSamplesPerSec=5.338407677414076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4034/5000 [14:04:13<2:57:08, 11.00s/it][2022-12-21 00:26:49,257] [INFO] [timer.py:197:stop] 0/8168, RunningAvgSamplesPerSec=5.879324068320064, CurrSamplesPerSec=5.33001766893835, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4035/5000 [14:04:24<2:57:12, 11.02s/it][2022-12-21 00:27:00,230] [INFO] [timer.py:197:stop] 0/8170, RunningAvgSamplesPerSec=5.879328832563151, CurrSamplesPerSec=5.327826364064929, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4036/5000 [14:04:35<2:56:48, 11.00s/it][2022-12-21 00:27:11,278] [INFO] [timer.py:197:stop] 0/8172, RunningAvgSamplesPerSec=5.879323737897583, CurrSamplesPerSec=5.297826113814562, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4037/5000 [14:04:46<2:56:49, 11.02s/it][2022-12-21 00:27:22,289] [INFO] [timer.py:197:stop] 0/8174, RunningAvgSamplesPerSec=5.879323514095325, CurrSamplesPerSec=5.340851027407004, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4038/5000 [14:04:57<2:56:37, 11.02s/it][2022-12-21 00:27:33,253] [INFO] [timer.py:197:stop] 0/8176, RunningAvgSamplesPerSec=5.879329714417915, CurrSamplesPerSec=5.343612097677564, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4039/5000 [14:05:08<2:56:11, 11.00s/it][2022-12-21 00:27:44,280] [INFO] [timer.py:197:stop] 0/8178, RunningAvgSamplesPerSec=5.879327269015308, CurrSamplesPerSec=5.326658348334905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4040/5000 [14:05:19<2:56:07, 11.01s/it][2022-12-21 00:27:55,347] [INFO] [logging.py:68:log_dist] [Rank 0] step=4090, skipped=5, lr=[2.0355555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:27:55,349] [INFO] [timer.py:197:stop] 0/8180, RunningAvgSamplesPerSec=5.879319206799394, CurrSamplesPerSec=5.294461080259542, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4041/5000 [14:05:30<2:56:14, 11.03s/it][2022-12-21 00:28:06,378] [INFO] [timer.py:197:stop] 0/8182, RunningAvgSamplesPerSec=5.879316601799705, CurrSamplesPerSec=5.31617552405933, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4042/5000 [14:05:41<2:56:04, 11.03s/it][2022-12-21 00:28:17,379] [INFO] [timer.py:197:stop] 0/8184, RunningAvgSamplesPerSec=5.8793174795792424, CurrSamplesPerSec=5.343528702823113, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4043/5000 [14:05:52<2:55:45, 11.02s/it][2022-12-21 00:28:28,416] [INFO] [timer.py:197:stop] 0/8186, RunningAvgSamplesPerSec=5.879313575772309, CurrSamplesPerSec=5.317720897653772, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4044/5000 [14:06:04<2:55:39, 11.02s/it][2022-12-21 00:28:39,425] [INFO] [timer.py:197:stop] 0/8188, RunningAvgSamplesPerSec=5.879313460482193, CurrSamplesPerSec=5.318454195355415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4045/5000 [14:06:15<2:55:24, 11.02s/it][2022-12-21 00:28:50,469] [INFO] [timer.py:197:stop] 0/8190, RunningAvgSamplesPerSec=5.87930858672184, CurrSamplesPerSec=5.3081671381953415, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4046/5000 [14:06:26<2:55:19, 11.03s/it][2022-12-21 00:29:01,511] [INFO] [timer.py:197:stop] 0/8192, RunningAvgSamplesPerSec=5.8793040365653395, CurrSamplesPerSec=5.304365259826807, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4047/5000 [14:06:37<2:55:13, 11.03s/it][2022-12-21 00:29:12,539] [INFO] [timer.py:197:stop] 0/8194, RunningAvgSamplesPerSec=5.879301497132757, CurrSamplesPerSec=5.34051207074838, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4048/5000 [14:06:48<2:55:00, 11.03s/it][2022-12-21 00:29:23,529] [INFO] [timer.py:197:stop] 0/8196, RunningAvgSamplesPerSec=5.879303695644854, CurrSamplesPerSec=5.357045332338806, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4049/5000 [14:06:59<2:54:38, 11.02s/it][2022-12-21 00:29:34,535] [INFO] [timer.py:197:stop] 0/8198, RunningAvgSamplesPerSec=5.879304068603482, CurrSamplesPerSec=5.320401365315319, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4050/5000 [14:07:10<2:54:23, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 2.0155555555555554e-06, 'epoch': 98.77}

 81%|████████  | 4050/5000 [14:07:10<2:54:23, 11.01s/it][2022-12-21 00:29:45,588] [INFO] [logging.py:68:log_dist] [Rank 0] step=4100, skipped=5, lr=[2.0133333333333337e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:29:45,589] [INFO] [timer.py:197:stop] 0/8200, RunningAvgSamplesPerSec=5.8792980656306755, CurrSamplesPerSec=5.3027859456804896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4051/5000 [14:07:21<2:54:24, 11.03s/it][2022-12-21 00:29:56,586] [INFO] [timer.py:197:stop] 0/8202, RunningAvgSamplesPerSec=5.879299507086128, CurrSamplesPerSec=5.367553163750655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4052/5000 [14:07:32<2:54:04, 11.02s/it][2022-12-21 00:30:07,603] [INFO] [timer.py:197:stop] 0/8204, RunningAvgSamplesPerSec=5.8792981870773104, CurrSamplesPerSec=5.330421343142767, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4053/5000 [14:07:43<2:53:53, 11.02s/it][2022-12-21 00:30:18,603] [INFO] [timer.py:197:stop] 0/8206, RunningAvgSamplesPerSec=5.879299247332314, CurrSamplesPerSec=5.3136585546421955, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4054/5000 [14:07:54<2:53:37, 11.01s/it][2022-12-21 00:30:29,649] [INFO] [timer.py:197:stop] 0/8208, RunningAvgSamplesPerSec=5.879294174526971, CurrSamplesPerSec=5.313461237818312, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4055/5000 [14:08:05<2:53:36, 11.02s/it][2022-12-21 00:30:40,708] [INFO] [timer.py:197:stop] 0/8210, RunningAvgSamplesPerSec=5.879287511915992, CurrSamplesPerSec=5.297798092531466, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4056/5000 [14:08:16<2:53:35, 11.03s/it][2022-12-21 00:30:51,780] [INFO] [timer.py:197:stop] 0/8212, RunningAvgSamplesPerSec=5.879279172400807, CurrSamplesPerSec=5.2937452406464445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4057/5000 [14:08:27<2:53:35, 11.04s/it][2022-12-21 00:31:02,790] [INFO] [timer.py:197:stop] 0/8214, RunningAvgSamplesPerSec=5.8792789194315, CurrSamplesPerSec=5.346420747331694, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4058/5000 [14:08:38<2:53:14, 11.03s/it][2022-12-21 00:31:13,849] [INFO] [timer.py:197:stop] 0/8216, RunningAvgSamplesPerSec=5.879272233361858, CurrSamplesPerSec=5.2794328545431055, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4059/5000 [14:08:49<2:53:10, 11.04s/it][2022-12-21 00:31:23,981] [INFO] [timer.py:197:stop] 0/8218, RunningAvgSamplesPerSec=5.879386338403868, CurrSamplesPerSec=5.298430107245878, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4060/5000 [14:09:04<3:12:02, 12.26s/it][2022-12-21 00:31:35,010] [INFO] [logging.py:68:log_dist] [Rank 0] step=4110, skipped=5, lr=[1.9911111111111113e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:31:35,012] [INFO] [timer.py:197:stop] 0/8220, RunningAvgSamplesPerSec=5.879383407324619, CurrSamplesPerSec=5.330578849817791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4061/5000 [14:09:15<3:06:01, 11.89s/it][2022-12-21 00:31:46,063] [INFO] [timer.py:197:stop] 0/8222, RunningAvgSamplesPerSec=5.8793774035216995, CurrSamplesPerSec=5.30016568135569, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████  | 4062/5000 [14:09:26<3:01:50, 11.63s/it][2022-12-21 00:31:57,096] [INFO] [timer.py:197:stop] 0/8224, RunningAvgSamplesPerSec=5.879374101981544, CurrSamplesPerSec=5.306484014885291, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4063/5000 [14:09:37<2:58:48, 11.45s/it][2022-12-21 00:32:08,137] [INFO] [timer.py:197:stop] 0/8226, RunningAvgSamplesPerSec=5.879369774379468, CurrSamplesPerSec=5.293551070163772, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4064/5000 [14:09:48<2:56:47, 11.33s/it][2022-12-21 00:32:19,139] [INFO] [timer.py:197:stop] 0/8228, RunningAvgSamplesPerSec=5.879370417958987, CurrSamplesPerSec=5.343381917183105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4065/5000 [14:09:59<2:54:59, 11.23s/it][2022-12-21 00:32:30,157] [INFO] [timer.py:197:stop] 0/8230, RunningAvgSamplesPerSec=5.879368973390279, CurrSamplesPerSec=5.317729114524873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4066/5000 [14:10:10<2:53:55, 11.17s/it][2022-12-21 00:32:41,200] [INFO] [timer.py:197:stop] 0/8232, RunningAvgSamplesPerSec=5.879364288039955, CurrSamplesPerSec=5.315493587498121, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4067/5000 [14:10:21<2:52:55, 11.12s/it][2022-12-21 00:32:52,212] [INFO] [timer.py:197:stop] 0/8234, RunningAvgSamplesPerSec=5.879363602434334, CurrSamplesPerSec=5.305016875195725, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4068/5000 [14:10:32<2:52:30, 11.11s/it][2022-12-21 00:33:03,337] [INFO] [timer.py:197:stop] 0/8236, RunningAvgSamplesPerSec=5.879348210411852, CurrSamplesPerSec=5.25656555054468, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4069/5000 [14:10:43<2:52:14, 11.10s/it][2022-12-21 00:33:14,317] [INFO] [timer.py:197:stop] 0/8238, RunningAvgSamplesPerSec=5.879351928844178, CurrSamplesPerSec=5.3506662453673215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4070/5000 [14:10:54<2:51:25, 11.06s/it][2022-12-21 00:33:25,334] [INFO] [logging.py:68:log_dist] [Rank 0] step=4120, skipped=5, lr=[1.968888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:33:25,336] [INFO] [timer.py:197:stop] 0/8240, RunningAvgSamplesPerSec=5.879350439568546, CurrSamplesPerSec=5.303852340653922, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4071/5000 [14:11:05<2:51:08, 11.05s/it][2022-12-21 00:33:36,379] [INFO] [timer.py:197:stop] 0/8242, RunningAvgSamplesPerSec=5.879345799719313, CurrSamplesPerSec=5.298996580782541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4072/5000 [14:11:16<2:50:53, 11.05s/it][2022-12-21 00:33:47,395] [INFO] [timer.py:197:stop] 0/8244, RunningAvgSamplesPerSec=5.879344885673582, CurrSamplesPerSec=5.319105061078207, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4073/5000 [14:11:27<2:50:40, 11.05s/it][2022-12-21 00:33:58,431] [INFO] [timer.py:197:stop] 0/8246, RunningAvgSamplesPerSec=5.879341245952769, CurrSamplesPerSec=5.325692225726671, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 81%|████████▏ | 4074/5000 [14:11:38<2:50:24, 11.04s/it][2022-12-21 00:34:09,464] [INFO] [timer.py:197:stop] 0/8248, RunningAvgSamplesPerSec=5.879338013802974, CurrSamplesPerSec=5.3221258299209016, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4075/5000 [14:11:50<2:50:07, 11.04s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.9600000000000003e-06, 'epoch': 99.39}

 82%|████████▏ | 4075/5000 [14:11:50<2:50:07, 11.04s/it][2022-12-21 00:34:20,516] [INFO] [timer.py:197:stop] 0/8250, RunningAvgSamplesPerSec=5.879332539982383, CurrSamplesPerSec=5.295277808145547, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4076/5000 [14:12:01<2:50:03, 11.04s/it][2022-12-21 00:34:31,583] [INFO] [timer.py:197:stop] 0/8252, RunningAvgSamplesPerSec=5.879324855602666, CurrSamplesPerSec=5.287708615297361, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4077/5000 [14:12:12<2:49:55, 11.05s/it][2022-12-21 00:34:42,582] [INFO] [timer.py:197:stop] 0/8254, RunningAvgSamplesPerSec=5.879325835855229, CurrSamplesPerSec=5.334560252616491, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4078/5000 [14:12:23<2:49:39, 11.04s/it][2022-12-21 00:34:53,600] [INFO] [timer.py:197:stop] 0/8256, RunningAvgSamplesPerSec=5.879324640207131, CurrSamplesPerSec=5.345490660586304, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4079/5000 [14:12:34<2:49:22, 11.03s/it][2022-12-21 00:35:04,637] [INFO] [timer.py:197:stop] 0/8258, RunningAvgSamplesPerSec=5.87932080682032, CurrSamplesPerSec=5.329677757907739, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4080/5000 [14:12:45<2:49:10, 11.03s/it][2022-12-21 00:35:15,671] [INFO] [logging.py:68:log_dist] [Rank 0] step=4130, skipped=5, lr=[1.9466666666666665e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:35:15,672] [INFO] [timer.py:197:stop] 0/8260, RunningAvgSamplesPerSec=5.87931726467643, CurrSamplesPerSec=5.325379700604006, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4081/5000 [14:12:56<2:48:53, 11.03s/it][2022-12-21 00:35:26,678] [INFO] [timer.py:197:stop] 0/8262, RunningAvgSamplesPerSec=5.879317527167356, CurrSamplesPerSec=5.331451232721348, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4082/5000 [14:13:07<2:48:35, 11.02s/it][2022-12-21 00:35:37,726] [INFO] [timer.py:197:stop] 0/8264, RunningAvgSamplesPerSec=5.879312260945233, CurrSamplesPerSec=5.2907326097628795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4083/5000 [14:13:18<2:48:28, 11.02s/it][2022-12-21 00:35:48,737] [INFO] [timer.py:197:stop] 0/8266, RunningAvgSamplesPerSec=5.8793119604205035, CurrSamplesPerSec=5.310472355955648, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4084/5000 [14:13:29<2:48:30, 11.04s/it][2022-12-21 00:35:59,757] [INFO] [timer.py:197:stop] 0/8268, RunningAvgSamplesPerSec=5.879310575610749, CurrSamplesPerSec=5.354943060774805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4085/5000 [14:13:40<2:48:09, 11.03s/it][2022-12-21 00:36:10,823] [INFO] [timer.py:197:stop] 0/8270, RunningAvgSamplesPerSec=5.879303357668457, CurrSamplesPerSec=5.30105661144027, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4086/5000 [14:13:51<2:48:10, 11.04s/it][2022-12-21 00:36:21,874] [INFO] [timer.py:197:stop] 0/8272, RunningAvgSamplesPerSec=5.87929781934806, CurrSamplesPerSec=5.315562846921397, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4087/5000 [14:14:02<2:47:47, 11.03s/it][2022-12-21 00:36:32,898] [INFO] [timer.py:197:stop] 0/8274, RunningAvgSamplesPerSec=5.879295931707599, CurrSamplesPerSec=5.293162771927432, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4088/5000 [14:14:13<2:47:49, 11.04s/it][2022-12-21 00:36:43,939] [INFO] [timer.py:197:stop] 0/8276, RunningAvgSamplesPerSec=5.8792915841199065, CurrSamplesPerSec=5.319952183831361, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4089/5000 [14:14:24<2:47:35, 11.04s/it][2022-12-21 00:36:54,982] [INFO] [timer.py:197:stop] 0/8278, RunningAvgSamplesPerSec=5.879287020187908, CurrSamplesPerSec=5.310920778571532, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4090/5000 [14:14:35<2:47:12, 11.03s/it][2022-12-21 00:37:05,982] [INFO] [logging.py:68:log_dist] [Rank 0] step=4140, skipped=5, lr=[1.9244444444444446e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:37:05,984] [INFO] [timer.py:197:stop] 0/8280, RunningAvgSamplesPerSec=5.879287821846341, CurrSamplesPerSec=5.304253109237295, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4091/5000 [14:14:46<2:47:15, 11.04s/it][2022-12-21 00:37:17,017] [INFO] [timer.py:197:stop] 0/8282, RunningAvgSamplesPerSec=5.879284536773077, CurrSamplesPerSec=5.340084770406883, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4092/5000 [14:14:57<2:46:46, 11.02s/it][2022-12-21 00:37:27,999] [INFO] [timer.py:197:stop] 0/8284, RunningAvgSamplesPerSec=5.879287845445242, CurrSamplesPerSec=5.332195101098891, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4093/5000 [14:15:08<2:46:21, 11.01s/it][2022-12-21 00:37:38,981] [INFO] [timer.py:197:stop] 0/8286, RunningAvgSamplesPerSec=5.87929115476188, CurrSamplesPerSec=5.322443038743695, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4094/5000 [14:15:19<2:46:09, 11.00s/it][2022-12-21 00:37:49,984] [INFO] [timer.py:197:stop] 0/8288, RunningAvgSamplesPerSec=5.879291947687337, CurrSamplesPerSec=5.321762658531498, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4095/5000 [14:15:30<2:45:55, 11.00s/it][2022-12-21 00:38:00,993] [INFO] [timer.py:197:stop] 0/8290, RunningAvgSamplesPerSec=5.879291927533418, CurrSamplesPerSec=5.307245065734518, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4096/5000 [14:15:41<2:45:55, 11.01s/it][2022-12-21 00:38:12,036] [INFO] [timer.py:197:stop] 0/8292, RunningAvgSamplesPerSec=5.8792872350944245, CurrSamplesPerSec=5.305441937117551, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4097/5000 [14:15:52<2:46:04, 11.03s/it][2022-12-21 00:38:23,122] [INFO] [timer.py:197:stop] 0/8294, RunningAvgSamplesPerSec=5.879277192539105, CurrSamplesPerSec=5.305190708272735, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4098/5000 [14:16:03<2:45:57, 11.04s/it][2022-12-21 00:38:34,132] [INFO] [timer.py:197:stop] 0/8296, RunningAvgSamplesPerSec=5.879276935406706, CurrSamplesPerSec=5.340533108185188, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4099/5000 [14:16:14<2:45:25, 11.02s/it][2022-12-21 00:38:45,118] [INFO] [timer.py:197:stop] 0/8298, RunningAvgSamplesPerSec=5.879280015720075, CurrSamplesPerSec=5.3210259168716565, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4100/5000 [14:16:25<2:45:10, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.9044444444444445e-06, 'epoch': 99.99}

 82%|████████▏ | 4100/5000 [14:16:25<2:45:10, 11.01s/it][2022-12-21 00:38:55,176] [INFO] [logging.py:68:log_dist] [Rank 0] step=4150, skipped=5, lr=[1.9022222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:38:55,178] [INFO] [timer.py:197:stop] 0/8300, RunningAvgSamplesPerSec=5.879402408551279, CurrSamplesPerSec=6.295832685812883, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 00:39:06,098] [INFO] [timer.py:197:stop] 0/8302, RunningAvgSamplesPerSec=5.879413597084445, CurrSamplesPerSec=5.374073305010129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4101/5000 [14:16:41<3:07:40, 12.53s/it][2022-12-21 00:39:17,119] [INFO] [timer.py:197:stop] 0/8304, RunningAvgSamplesPerSec=5.879411955153533, CurrSamplesPerSec=5.337322037043201, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4102/5000 [14:16:52<3:00:42, 12.07s/it][2022-12-21 00:39:28,195] [INFO] [timer.py:197:stop] 0/8306, RunningAvgSamplesPerSec=5.87940329020361, CurrSamplesPerSec=5.281855554900549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4103/5000 [14:17:03<2:56:01, 11.77s/it][2022-12-21 00:39:39,195] [INFO] [timer.py:197:stop] 0/8308, RunningAvgSamplesPerSec=5.879404274059555, CurrSamplesPerSec=5.344067411403323, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4104/5000 [14:17:14<2:52:21, 11.54s/it][2022-12-21 00:39:50,224] [INFO] [timer.py:197:stop] 0/8310, RunningAvgSamplesPerSec=5.8794015580083, CurrSamplesPerSec=5.3158142169505185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4105/5000 [14:17:25<2:49:52, 11.39s/it][2022-12-21 00:40:01,195] [INFO] [timer.py:197:stop] 0/8312, RunningAvgSamplesPerSec=5.879406500998291, CurrSamplesPerSec=5.342292126509466, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4106/5000 [14:17:36<2:47:49, 11.26s/it][2022-12-21 00:40:12,226] [INFO] [timer.py:197:stop] 0/8314, RunningAvgSamplesPerSec=5.879403612138365, CurrSamplesPerSec=5.31175310370014, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4107/5000 [14:17:47<2:46:35, 11.19s/it][2022-12-21 00:40:23,193] [INFO] [timer.py:197:stop] 0/8316, RunningAvgSamplesPerSec=5.879408989600911, CurrSamplesPerSec=5.333709107488297, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4108/5000 [14:17:58<2:45:24, 11.13s/it][2022-12-21 00:40:34,182] [INFO] [timer.py:197:stop] 0/8318, RunningAvgSamplesPerSec=5.879411655495076, CurrSamplesPerSec=5.326075800603458, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4109/5000 [14:18:09<2:44:36, 11.08s/it][2022-12-21 00:40:45,214] [INFO] [logging.py:68:log_dist] [Rank 0] step=4160, skipped=5, lr=[1.8800000000000002e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:40:45,216] [INFO] [timer.py:197:stop] 0/8320, RunningAvgSamplesPerSec=5.879408422395517, CurrSamplesPerSec=5.325705961614112, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4110/5000 [14:18:20<2:44:11, 11.07s/it][2022-12-21 00:40:56,193] [INFO] [timer.py:197:stop] 0/8322, RunningAvgSamplesPerSec=5.879412636950251, CurrSamplesPerSec=5.368000329236527, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4111/5000 [14:18:31<2:43:36, 11.04s/it][2022-12-21 00:41:07,164] [INFO] [timer.py:197:stop] 0/8324, RunningAvgSamplesPerSec=5.8794175040949215, CurrSamplesPerSec=5.344241472382163, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4112/5000 [14:18:42<2:43:06, 11.02s/it][2022-12-21 00:41:18,164] [INFO] [timer.py:197:stop] 0/8326, RunningAvgSamplesPerSec=5.879418835814911, CurrSamplesPerSec=5.30564893568786, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4113/5000 [14:18:53<2:42:49, 11.01s/it][2022-12-21 00:41:29,172] [INFO] [timer.py:197:stop] 0/8328, RunningAvgSamplesPerSec=5.87941902936405, CurrSamplesPerSec=5.345746572512703, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4114/5000 [14:19:04<2:42:36, 11.01s/it][2022-12-21 00:41:40,191] [INFO] [timer.py:197:stop] 0/8330, RunningAvgSamplesPerSec=5.879417660891862, CurrSamplesPerSec=5.319440462436909, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4115/5000 [14:19:15<2:42:27, 11.01s/it][2022-12-21 00:41:51,211] [INFO] [timer.py:197:stop] 0/8332, RunningAvgSamplesPerSec=5.879416234889478, CurrSamplesPerSec=5.3203868131770555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4116/5000 [14:19:26<2:42:18, 11.02s/it][2022-12-21 00:42:02,285] [INFO] [timer.py:197:stop] 0/8334, RunningAvgSamplesPerSec=5.879407671709497, CurrSamplesPerSec=5.290706331894598, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4117/5000 [14:19:37<2:42:22, 11.03s/it][2022-12-21 00:42:13,297] [INFO] [timer.py:197:stop] 0/8336, RunningAvgSamplesPerSec=5.879407328551862, CurrSamplesPerSec=5.326157806147749, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4118/5000 [14:19:48<2:42:05, 11.03s/it][2022-12-21 00:42:24,320] [INFO] [timer.py:197:stop] 0/8338, RunningAvgSamplesPerSec=5.87940536708284, CurrSamplesPerSec=5.319073863102032, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4119/5000 [14:19:59<2:41:53, 11.03s/it][2022-12-21 00:42:35,345] [INFO] [logging.py:68:log_dist] [Rank 0] step=4170, skipped=5, lr=[1.8577777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:42:35,347] [INFO] [timer.py:197:stop] 0/8340, RunningAvgSamplesPerSec=5.8794029038482085, CurrSamplesPerSec=5.325339554666794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4120/5000 [14:20:10<2:41:42, 11.03s/it][2022-12-21 00:42:46,377] [INFO] [timer.py:197:stop] 0/8342, RunningAvgSamplesPerSec=5.879400008136403, CurrSamplesPerSec=5.328734234667739, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4121/5000 [14:20:21<2:41:32, 11.03s/it][2022-12-21 00:42:57,410] [INFO] [timer.py:197:stop] 0/8344, RunningAvgSamplesPerSec=5.879396917892875, CurrSamplesPerSec=5.3063845719258165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4122/5000 [14:20:33<2:41:23, 11.03s/it][2022-12-21 00:43:08,430] [INFO] [timer.py:197:stop] 0/8346, RunningAvgSamplesPerSec=5.879395533186891, CurrSamplesPerSec=5.320621555208884, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4123/5000 [14:20:44<2:41:10, 11.03s/it][2022-12-21 00:43:19,392] [INFO] [timer.py:197:stop] 0/8348, RunningAvgSamplesPerSec=5.879401568406628, CurrSamplesPerSec=5.337023637625309, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▏ | 4124/5000 [14:20:54<2:40:42, 11.01s/it][2022-12-21 00:43:30,357] [INFO] [timer.py:197:stop] 0/8350, RunningAvgSamplesPerSec=5.879407324596102, CurrSamplesPerSec=5.357073983872946, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 82%|████████▎ | 4125/5000 [14:21:05<2:40:20, 10.99s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.8466666666666668e-06, 'epoch': 100.6}

 82%|████████▎ | 4125/5000 [14:21:05<2:40:20, 10.99s/it][2022-12-21 00:43:41,357] [INFO] [timer.py:197:stop] 0/8352, RunningAvgSamplesPerSec=5.879408571046578, CurrSamplesPerSec=5.328778663116946, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4126/5000 [14:21:16<2:40:10, 11.00s/it][2022-12-21 00:43:52,343] [INFO] [timer.py:197:stop] 0/8354, RunningAvgSamplesPerSec=5.879411531665187, CurrSamplesPerSec=5.34060132162026, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4127/5000 [14:21:27<2:39:57, 10.99s/it][2022-12-21 00:44:03,407] [INFO] [timer.py:197:stop] 0/8356, RunningAvgSamplesPerSec=5.879404435660339, CurrSamplesPerSec=5.314817297785836, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4128/5000 [14:21:39<2:40:04, 11.01s/it][2022-12-21 00:44:14,473] [INFO] [timer.py:197:stop] 0/8358, RunningAvgSamplesPerSec=5.879397228381717, CurrSamplesPerSec=5.28983701532112, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4129/5000 [14:21:50<2:40:06, 11.03s/it][2022-12-21 00:44:25,495] [INFO] [logging.py:68:log_dist] [Rank 0] step=4180, skipped=5, lr=[1.8355555555555557e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:44:25,496] [INFO] [timer.py:197:stop] 0/8360, RunningAvgSamplesPerSec=5.879395419203502, CurrSamplesPerSec=5.30878042133571, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4130/5000 [14:22:01<2:39:54, 11.03s/it][2022-12-21 00:44:36,513] [INFO] [timer.py:197:stop] 0/8362, RunningAvgSamplesPerSec=5.879394298349533, CurrSamplesPerSec=5.313073168059205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4131/5000 [14:22:12<2:39:40, 11.02s/it][2022-12-21 00:44:47,583] [INFO] [timer.py:197:stop] 0/8364, RunningAvgSamplesPerSec=5.8793861481877085, CurrSamplesPerSec=5.285998259474284, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4132/5000 [14:22:23<2:39:41, 11.04s/it][2022-12-21 00:44:58,592] [INFO] [timer.py:197:stop] 0/8366, RunningAvgSamplesPerSec=5.879386087523323, CurrSamplesPerSec=5.334557496291045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4133/5000 [14:22:34<2:39:22, 11.03s/it][2022-12-21 00:45:09,601] [INFO] [timer.py:197:stop] 0/8368, RunningAvgSamplesPerSec=5.879386337783563, CurrSamplesPerSec=5.3365389688647715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4134/5000 [14:22:45<2:39:06, 11.02s/it][2022-12-21 00:45:20,549] [INFO] [timer.py:197:stop] 0/8370, RunningAvgSamplesPerSec=5.879394237605077, CurrSamplesPerSec=5.354860807368847, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4135/5000 [14:22:56<2:38:35, 11.00s/it][2022-12-21 00:45:31,575] [INFO] [timer.py:197:stop] 0/8372, RunningAvgSamplesPerSec=5.879391975846696, CurrSamplesPerSec=5.295259632686927, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4136/5000 [14:23:07<2:38:31, 11.01s/it][2022-12-21 00:45:41,960] [INFO] [stage_1_and_2.py:1765:step] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072.0, reducing to 131072.0
[2022-12-21 00:45:41,962] [INFO] [timer.py:197:stop] 0/8374, RunningAvgSamplesPerSec=5.879472058375945, CurrSamplesPerSec=5.935283442883516, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4137/5000 [14:23:17<2:35:39, 10.82s/it][2022-12-21 00:45:52,281] [INFO] [stage_1_and_2.py:1765:step] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2022-12-21 00:45:52,283] [INFO] [timer.py:197:stop] 0/8376, RunningAvgSamplesPerSec=5.879560693339627, CurrSamplesPerSec=5.967002035759316, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4138/5000 [14:23:27<2:33:18, 10.67s/it][2022-12-21 00:46:03,287] [INFO] [timer.py:197:stop] 0/8378, RunningAvgSamplesPerSec=5.879561182430551, CurrSamplesPerSec=5.341895156746999, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4139/5000 [14:23:38<2:34:34, 10.77s/it][2022-12-21 00:46:14,338] [INFO] [logging.py:68:log_dist] [Rank 0] step=4190, skipped=7, lr=[1.817777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:46:14,340] [INFO] [timer.py:197:stop] 0/8380, RunningAvgSamplesPerSec=5.879555372610727, CurrSamplesPerSec=5.308146145046857, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4140/5000 [14:23:49<2:35:35, 10.86s/it][2022-12-21 00:46:25,344] [INFO] [timer.py:197:stop] 0/8382, RunningAvgSamplesPerSec=5.879555805723497, CurrSamplesPerSec=5.324771871398592, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4141/5000 [14:24:00<2:36:03, 10.90s/it][2022-12-21 00:46:35,479] [INFO] [timer.py:197:stop] 0/8384, RunningAvgSamplesPerSec=5.879667104371806, CurrSamplesPerSec=5.316851317016506, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4142/5000 [14:24:16<2:53:51, 12.16s/it][2022-12-21 00:46:46,506] [INFO] [timer.py:197:stop] 0/8386, RunningAvgSamplesPerSec=5.879664441580888, CurrSamplesPerSec=5.327641105094203, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4143/5000 [14:24:27<2:48:42, 11.81s/it][2022-12-21 00:46:57,500] [INFO] [timer.py:197:stop] 0/8388, RunningAvgSamplesPerSec=5.8796660646731045, CurrSamplesPerSec=5.33418711446512, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4144/5000 [14:24:38<2:45:03, 11.57s/it][2022-12-21 00:47:08,514] [INFO] [timer.py:197:stop] 0/8390, RunningAvgSamplesPerSec=5.879665109295008, CurrSamplesPerSec=5.328179999145698, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4145/5000 [14:24:49<2:42:24, 11.40s/it][2022-12-21 00:47:19,543] [INFO] [timer.py:197:stop] 0/8392, RunningAvgSamplesPerSec=5.879662307391213, CurrSamplesPerSec=5.298306285736392, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4146/5000 [14:25:00<2:40:44, 11.29s/it][2022-12-21 00:47:30,589] [INFO] [timer.py:197:stop] 0/8394, RunningAvgSamplesPerSec=5.879657297504832, CurrSamplesPerSec=5.301620505539521, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4147/5000 [14:25:11<2:39:30, 11.22s/it][2022-12-21 00:47:41,608] [INFO] [timer.py:197:stop] 0/8396, RunningAvgSamplesPerSec=5.8796558149728195, CurrSamplesPerSec=5.327340192469398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4148/5000 [14:25:22<2:38:21, 11.15s/it][2022-12-21 00:47:52,588] [INFO] [timer.py:197:stop] 0/8398, RunningAvgSamplesPerSec=5.879659317862077, CurrSamplesPerSec=5.339260110504903, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4149/5000 [14:25:33<2:37:22, 11.10s/it][2022-12-21 00:48:03,544] [INFO] [logging.py:68:log_dist] [Rank 0] step=4200, skipped=7, lr=[1.7955555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:48:03,546] [INFO] [timer.py:197:stop] 0/8400, RunningAvgSamplesPerSec=5.8796657087285595, CurrSamplesPerSec=5.34709019270009, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4150/5000 [14:25:44<2:36:51, 11.07s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.7955555555555556e-06, 'epoch': 101.22}

 83%|████████▎ | 4150/5000 [14:25:44<2:36:51, 11.07s/it][2022-12-21 00:48:14,590] [INFO] [timer.py:197:stop] 0/8402, RunningAvgSamplesPerSec=5.879661096910081, CurrSamplesPerSec=5.324668995723134, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4151/5000 [14:25:55<2:36:21, 11.05s/it][2022-12-21 00:48:25,613] [INFO] [timer.py:197:stop] 0/8404, RunningAvgSamplesPerSec=5.879659187971538, CurrSamplesPerSec=5.301951820472288, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4152/5000 [14:26:06<2:35:57, 11.03s/it][2022-12-21 00:48:36,568] [INFO] [timer.py:197:stop] 0/8406, RunningAvgSamplesPerSec=5.879665717066662, CurrSamplesPerSec=5.339894196532439, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4153/5000 [14:26:17<2:35:40, 11.03s/it][2022-12-21 00:48:47,620] [INFO] [timer.py:197:stop] 0/8408, RunningAvgSamplesPerSec=5.879660047403878, CurrSamplesPerSec=5.305370424648294, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4154/5000 [14:26:28<2:35:35, 11.03s/it][2022-12-21 00:48:58,670] [INFO] [timer.py:197:stop] 0/8410, RunningAvgSamplesPerSec=5.879654527062707, CurrSamplesPerSec=5.305006600738334, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4155/5000 [14:26:39<2:35:24, 11.03s/it][2022-12-21 00:49:09,677] [INFO] [timer.py:197:stop] 0/8412, RunningAvgSamplesPerSec=5.879654713144678, CurrSamplesPerSec=5.330944284215078, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4156/5000 [14:26:50<2:34:54, 11.01s/it][2022-12-21 00:49:20,638] [INFO] [timer.py:197:stop] 0/8414, RunningAvgSamplesPerSec=5.879660606663472, CurrSamplesPerSec=5.331174453145186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4157/5000 [14:27:01<2:34:45, 11.02s/it][2022-12-21 00:49:31,663] [INFO] [timer.py:197:stop] 0/8416, RunningAvgSamplesPerSec=5.879658297128644, CurrSamplesPerSec=5.326898929529991, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4158/5000 [14:27:12<2:34:36, 11.02s/it][2022-12-21 00:49:42,656] [INFO] [timer.py:197:stop] 0/8418, RunningAvgSamplesPerSec=5.879660158556979, CurrSamplesPerSec=5.351426580828682, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4159/5000 [14:27:23<2:34:17, 11.01s/it][2022-12-21 00:49:53,670] [INFO] [logging.py:68:log_dist] [Rank 0] step=4210, skipped=7, lr=[1.7733333333333336e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:49:53,672] [INFO] [timer.py:197:stop] 0/8420, RunningAvgSamplesPerSec=5.879658973887545, CurrSamplesPerSec=5.326159285652409, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4160/5000 [14:27:34<2:34:02, 11.00s/it][2022-12-21 00:50:04,641] [INFO] [timer.py:197:stop] 0/8422, RunningAvgSamplesPerSec=5.8796637542619985, CurrSamplesPerSec=5.343947405302789, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4161/5000 [14:27:45<2:33:42, 10.99s/it][2022-12-21 00:50:15,600] [INFO] [timer.py:197:stop] 0/8424, RunningAvgSamplesPerSec=5.87966984986061, CurrSamplesPerSec=5.351892617094271, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4162/5000 [14:27:56<2:33:31, 10.99s/it][2022-12-21 00:50:26,640] [INFO] [timer.py:197:stop] 0/8426, RunningAvgSamplesPerSec=5.879665595666846, CurrSamplesPerSec=5.308629659383269, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4163/5000 [14:28:07<2:33:34, 11.01s/it][2022-12-21 00:50:37,680] [INFO] [timer.py:197:stop] 0/8428, RunningAvgSamplesPerSec=5.879661471458285, CurrSamplesPerSec=5.319170620180138, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4164/5000 [14:28:18<2:33:30, 11.02s/it][2022-12-21 00:50:48,662] [INFO] [timer.py:197:stop] 0/8430, RunningAvgSamplesPerSec=5.879664583778336, CurrSamplesPerSec=5.364266220145827, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4165/5000 [14:28:29<2:33:09, 11.01s/it][2022-12-21 00:50:59,717] [INFO] [timer.py:197:stop] 0/8432, RunningAvgSamplesPerSec=5.879658405520166, CurrSamplesPerSec=5.297931509951434, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4166/5000 [14:28:40<2:33:14, 11.02s/it][2022-12-21 00:51:10,748] [INFO] [timer.py:197:stop] 0/8434, RunningAvgSamplesPerSec=5.879655185345949, CurrSamplesPerSec=5.330481677261155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4167/5000 [14:28:51<2:32:53, 11.01s/it][2022-12-21 00:51:21,689] [INFO] [timer.py:197:stop] 0/8436, RunningAvgSamplesPerSec=5.879663793119303, CurrSamplesPerSec=5.3719054037152105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4168/5000 [14:29:02<2:32:22, 10.99s/it][2022-12-21 00:51:32,656] [INFO] [timer.py:197:stop] 0/8438, RunningAvgSamplesPerSec=5.879668963875951, CurrSamplesPerSec=5.340900333778097, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4169/5000 [14:29:13<2:32:06, 10.98s/it][2022-12-21 00:51:43,595] [INFO] [logging.py:68:log_dist] [Rank 0] step=4220, skipped=7, lr=[1.7511111111111113e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:51:43,597] [INFO] [timer.py:197:stop] 0/8440, RunningAvgSamplesPerSec=5.879677441937503, CurrSamplesPerSec=5.365672364068832, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4170/5000 [14:29:24<2:31:51, 10.98s/it][2022-12-21 00:51:54,613] [INFO] [timer.py:197:stop] 0/8442, RunningAvgSamplesPerSec=5.8796763080729, CurrSamplesPerSec=5.32049711615216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4171/5000 [14:29:35<2:31:35, 10.97s/it][2022-12-21 00:52:05,542] [INFO] [timer.py:197:stop] 0/8444, RunningAvgSamplesPerSec=5.87968642508094, CurrSamplesPerSec=5.345220084859095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4172/5000 [14:29:46<2:31:23, 10.97s/it][2022-12-21 00:52:16,533] [INFO] [timer.py:197:stop] 0/8446, RunningAvgSamplesPerSec=5.879688416876027, CurrSamplesPerSec=5.325480702007211, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4173/5000 [14:29:57<2:31:25, 10.99s/it][2022-12-21 00:52:27,553] [INFO] [timer.py:197:stop] 0/8448, RunningAvgSamplesPerSec=5.879686781714583, CurrSamplesPerSec=5.325869107110985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 83%|████████▎ | 4174/5000 [14:30:08<2:31:19, 10.99s/it][2022-12-21 00:52:38,586] [INFO] [timer.py:197:stop] 0/8450, RunningAvgSamplesPerSec=5.879683571104116, CurrSamplesPerSec=5.304717045152238, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4175/5000 [14:30:19<2:31:20, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.74e-06, 'epoch': 101.82}

 84%|████████▎ | 4175/5000 [14:30:19<2:31:20, 11.01s/it][2022-12-21 00:52:49,609] [INFO] [timer.py:197:stop] 0/8452, RunningAvgSamplesPerSec=5.8796819432311365, CurrSamplesPerSec=5.320857583579354, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4176/5000 [14:30:30<2:31:02, 11.00s/it][2022-12-21 00:53:00,560] [INFO] [timer.py:197:stop] 0/8454, RunningAvgSamplesPerSec=5.879689247913942, CurrSamplesPerSec=5.343387022635213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4177/5000 [14:30:41<2:30:41, 10.99s/it][2022-12-21 00:53:11,540] [INFO] [timer.py:197:stop] 0/8456, RunningAvgSamplesPerSec=5.87969276761876, CurrSamplesPerSec=5.322449159578507, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4178/5000 [14:30:52<2:30:29, 10.98s/it][2022-12-21 00:53:22,531] [INFO] [timer.py:197:stop] 0/8458, RunningAvgSamplesPerSec=5.879695038829483, CurrSamplesPerSec=5.316233219834995, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4179/5000 [14:31:03<2:30:32, 11.00s/it][2022-12-21 00:53:33,566] [INFO] [logging.py:68:log_dist] [Rank 0] step=4230, skipped=7, lr=[1.728888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:53:33,568] [INFO] [timer.py:197:stop] 0/8460, RunningAvgSamplesPerSec=5.879691356172742, CurrSamplesPerSec=5.320038218355553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4180/5000 [14:31:14<2:30:19, 11.00s/it][2022-12-21 00:53:44,527] [INFO] [timer.py:197:stop] 0/8462, RunningAvgSamplesPerSec=5.879698635061717, CurrSamplesPerSec=5.356279124412723, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4181/5000 [14:31:25<2:30:09, 11.00s/it][2022-12-21 00:53:55,518] [INFO] [timer.py:197:stop] 0/8464, RunningAvgSamplesPerSec=5.879700830165782, CurrSamplesPerSec=5.359898732858876, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4182/5000 [14:31:36<2:29:47, 10.99s/it][2022-12-21 00:54:05,592] [INFO] [timer.py:197:stop] 0/8466, RunningAvgSamplesPerSec=5.879818779463994, CurrSamplesPerSec=6.277921683682717, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 00:54:16,609] [INFO] [timer.py:197:stop] 0/8468, RunningAvgSamplesPerSec=5.879817229961785, CurrSamplesPerSec=5.342756998445471, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4183/5000 [14:31:52<2:50:46, 12.54s/it][2022-12-21 00:54:27,614] [INFO] [timer.py:197:stop] 0/8470, RunningAvgSamplesPerSec=5.879817488838075, CurrSamplesPerSec=5.3357682213089275, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4184/5000 [14:32:03<2:44:17, 12.08s/it][2022-12-21 00:54:38,627] [INFO] [timer.py:197:stop] 0/8472, RunningAvgSamplesPerSec=5.879817041016966, CurrSamplesPerSec=5.331419042648102, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4185/5000 [14:32:14<2:39:44, 11.76s/it][2022-12-21 00:54:49,592] [INFO] [timer.py:197:stop] 0/8474, RunningAvgSamplesPerSec=5.879822402718826, CurrSamplesPerSec=5.336799966058936, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4186/5000 [14:32:25<2:36:18, 11.52s/it][2022-12-21 00:55:00,560] [INFO] [timer.py:197:stop] 0/8476, RunningAvgSamplesPerSec=5.87982740175581, CurrSamplesPerSec=5.317093540794607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▎ | 4187/5000 [14:32:36<2:33:52, 11.36s/it][2022-12-21 00:55:11,612] [INFO] [timer.py:197:stop] 0/8478, RunningAvgSamplesPerSec=5.879821799684799, CurrSamplesPerSec=5.316793607824019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4188/5000 [14:32:47<2:32:26, 11.26s/it][2022-12-21 00:55:22,627] [INFO] [logging.py:68:log_dist] [Rank 0] step=4240, skipped=7, lr=[1.7066666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:55:22,629] [INFO] [timer.py:197:stop] 0/8480, RunningAvgSamplesPerSec=5.879820597810602, CurrSamplesPerSec=5.312367636312211, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4189/5000 [14:32:58<2:31:15, 11.19s/it][2022-12-21 00:55:33,613] [INFO] [timer.py:197:stop] 0/8482, RunningAvgSamplesPerSec=5.879823615105404, CurrSamplesPerSec=5.323729567729799, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4190/5000 [14:33:09<2:30:13, 11.13s/it][2022-12-21 00:55:44,640] [INFO] [timer.py:197:stop] 0/8484, RunningAvgSamplesPerSec=5.879821465635901, CurrSamplesPerSec=5.348264203464134, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4191/5000 [14:33:20<2:29:38, 11.10s/it][2022-12-21 00:55:55,637] [INFO] [timer.py:197:stop] 0/8486, RunningAvgSamplesPerSec=5.879822668531959, CurrSamplesPerSec=5.3570489672179535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4192/5000 [14:33:31<2:29:02, 11.07s/it][2022-12-21 00:56:06,639] [INFO] [timer.py:197:stop] 0/8488, RunningAvgSamplesPerSec=5.879823411453967, CurrSamplesPerSec=5.307147692765277, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4193/5000 [14:33:42<2:28:35, 11.05s/it][2022-12-21 00:56:17,626] [INFO] [timer.py:197:stop] 0/8490, RunningAvgSamplesPerSec=5.879826007732794, CurrSamplesPerSec=5.342149661371552, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4194/5000 [14:33:53<2:28:09, 11.03s/it][2022-12-21 00:56:28,661] [INFO] [timer.py:197:stop] 0/8492, RunningAvgSamplesPerSec=5.879822598078155, CurrSamplesPerSec=5.351265706202805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4195/5000 [14:34:04<2:28:00, 11.03s/it][2022-12-21 00:56:39,667] [INFO] [timer.py:197:stop] 0/8494, RunningAvgSamplesPerSec=5.879822977805749, CurrSamplesPerSec=5.306105564112377, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4196/5000 [14:34:15<2:27:43, 11.02s/it][2022-12-21 00:56:50,730] [INFO] [timer.py:197:stop] 0/8496, RunningAvgSamplesPerSec=5.879816157781824, CurrSamplesPerSec=5.2906237460079195, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4197/5000 [14:34:26<2:27:41, 11.04s/it][2022-12-21 00:57:01,723] [INFO] [timer.py:197:stop] 0/8498, RunningAvgSamplesPerSec=5.879818186033205, CurrSamplesPerSec=5.311964379236667, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4198/5000 [14:34:37<2:27:20, 11.02s/it][2022-12-21 00:57:12,741] [INFO] [logging.py:68:log_dist] [Rank 0] step=4250, skipped=7, lr=[1.6844444444444447e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:57:12,742] [INFO] [timer.py:197:stop] 0/8500, RunningAvgSamplesPerSec=5.8798167700243456, CurrSamplesPerSec=5.341754200961612, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4199/5000 [14:34:48<2:27:08, 11.02s/it][2022-12-21 00:57:23,765] [INFO] [timer.py:197:stop] 0/8502, RunningAvgSamplesPerSec=5.879814940640328, CurrSamplesPerSec=5.3462516552967605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4200/5000 [14:34:59<2:26:57, 11.02s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.6822222222222224e-06, 'epoch': 102.43}

 84%|████████▍ | 4200/5000 [14:34:59<2:26:57, 11.02s/it][2022-12-21 00:57:34,733] [INFO] [timer.py:197:stop] 0/8504, RunningAvgSamplesPerSec=5.879820030056155, CurrSamplesPerSec=5.352814045747434, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4201/5000 [14:35:10<2:26:33, 11.01s/it][2022-12-21 00:57:45,722] [INFO] [timer.py:197:stop] 0/8506, RunningAvgSamplesPerSec=5.879822525254739, CurrSamplesPerSec=5.356367834255162, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4202/5000 [14:35:21<2:26:18, 11.00s/it][2022-12-21 00:57:56,755] [INFO] [timer.py:197:stop] 0/8508, RunningAvgSamplesPerSec=5.8798192972944685, CurrSamplesPerSec=5.329762202491029, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4203/5000 [14:35:32<2:26:15, 11.01s/it][2022-12-21 00:58:07,791] [INFO] [timer.py:197:stop] 0/8510, RunningAvgSamplesPerSec=5.879815689142933, CurrSamplesPerSec=5.313200204709328, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4204/5000 [14:35:43<2:26:10, 11.02s/it][2022-12-21 00:58:18,871] [INFO] [timer.py:197:stop] 0/8512, RunningAvgSamplesPerSec=5.879806381579813, CurrSamplesPerSec=5.282216833830814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4205/5000 [14:35:54<2:26:14, 11.04s/it][2022-12-21 00:58:29,942] [INFO] [timer.py:197:stop] 0/8514, RunningAvgSamplesPerSec=5.879798231727194, CurrSamplesPerSec=5.307425341071713, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4206/5000 [14:36:05<2:26:11, 11.05s/it][2022-12-21 00:58:40,971] [INFO] [timer.py:197:stop] 0/8516, RunningAvgSamplesPerSec=5.879795480167354, CurrSamplesPerSec=5.318413310792897, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4207/5000 [14:36:16<2:25:55, 11.04s/it][2022-12-21 00:58:51,968] [INFO] [timer.py:197:stop] 0/8518, RunningAvgSamplesPerSec=5.879796738958702, CurrSamplesPerSec=5.334033210184575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4208/5000 [14:36:27<2:25:34, 11.03s/it][2022-12-21 00:59:03,042] [INFO] [logging.py:68:log_dist] [Rank 0] step=4260, skipped=7, lr=[1.6622222222222224e-06], mom=[[0.9, 0.999]]
[2022-12-21 00:59:03,044] [INFO] [timer.py:197:stop] 0/8520, RunningAvgSamplesPerSec=5.879788061594417, CurrSamplesPerSec=5.29999092170661, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4209/5000 [14:36:38<2:25:34, 11.04s/it][2022-12-21 00:59:14,018] [INFO] [timer.py:197:stop] 0/8522, RunningAvgSamplesPerSec=5.879792136754514, CurrSamplesPerSec=5.348200269462686, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4210/5000 [14:36:49<2:25:07, 11.02s/it][2022-12-21 00:59:25,053] [INFO] [timer.py:197:stop] 0/8524, RunningAvgSamplesPerSec=5.8797887582585355, CurrSamplesPerSec=5.326470845372696, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4211/5000 [14:37:00<2:24:59, 11.03s/it][2022-12-21 00:59:36,039] [INFO] [timer.py:197:stop] 0/8526, RunningAvgSamplesPerSec=5.879791592814843, CurrSamplesPerSec=5.347745103554848, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4212/5000 [14:37:11<2:24:38, 11.01s/it][2022-12-21 00:59:47,094] [INFO] [timer.py:197:stop] 0/8528, RunningAvgSamplesPerSec=5.879785776145396, CurrSamplesPerSec=5.314856232922785, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4213/5000 [14:37:22<2:24:37, 11.03s/it][2022-12-21 00:59:58,091] [INFO] [timer.py:197:stop] 0/8530, RunningAvgSamplesPerSec=5.87978692428597, CurrSamplesPerSec=5.314828452118117, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4214/5000 [14:37:33<2:24:19, 11.02s/it][2022-12-21 01:00:09,127] [INFO] [timer.py:197:stop] 0/8532, RunningAvgSamplesPerSec=5.879783326940305, CurrSamplesPerSec=5.322858654294942, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4215/5000 [14:37:44<2:24:13, 11.02s/it][2022-12-21 01:00:20,130] [INFO] [timer.py:197:stop] 0/8534, RunningAvgSamplesPerSec=5.879783766407282, CurrSamplesPerSec=5.324480576662671, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4216/5000 [14:37:55<2:23:57, 11.02s/it][2022-12-21 01:00:31,146] [INFO] [timer.py:197:stop] 0/8536, RunningAvgSamplesPerSec=5.879782534957491, CurrSamplesPerSec=5.3440806038921265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4217/5000 [14:38:06<2:23:46, 11.02s/it][2022-12-21 01:00:42,204] [INFO] [timer.py:197:stop] 0/8538, RunningAvgSamplesPerSec=5.879776110843483, CurrSamplesPerSec=5.2778996891090655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4218/5000 [14:38:17<2:23:44, 11.03s/it][2022-12-21 01:00:53,248] [INFO] [logging.py:68:log_dist] [Rank 0] step=4270, skipped=7, lr=[1.6400000000000002e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:00:53,250] [INFO] [timer.py:197:stop] 0/8540, RunningAvgSamplesPerSec=5.879771184666082, CurrSamplesPerSec=5.311778960400536, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4219/5000 [14:38:28<2:23:37, 11.03s/it][2022-12-21 01:01:04,276] [INFO] [timer.py:197:stop] 0/8542, RunningAvgSamplesPerSec=5.879768796328558, CurrSamplesPerSec=5.307556515341538, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4220/5000 [14:38:39<2:23:24, 11.03s/it][2022-12-21 01:01:15,285] [INFO] [timer.py:197:stop] 0/8544, RunningAvgSamplesPerSec=5.879768664269369, CurrSamplesPerSec=5.342666399489019, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4221/5000 [14:38:50<2:23:08, 11.02s/it][2022-12-21 01:01:26,303] [INFO] [timer.py:197:stop] 0/8546, RunningAvgSamplesPerSec=5.87976740937338, CurrSamplesPerSec=5.346698899311096, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4222/5000 [14:39:01<2:22:55, 11.02s/it][2022-12-21 01:01:37,335] [INFO] [timer.py:197:stop] 0/8548, RunningAvgSamplesPerSec=5.879764222048964, CurrSamplesPerSec=5.326062274158359, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4223/5000 [14:39:12<2:22:46, 11.03s/it][2022-12-21 01:01:47,419] [INFO] [timer.py:197:stop] 0/8550, RunningAvgSamplesPerSec=5.879879769339955, CurrSamplesPerSec=5.356211151148356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4224/5000 [14:39:27<2:38:10, 12.23s/it][2022-12-21 01:01:58,414] [INFO] [timer.py:197:stop] 0/8552, RunningAvgSamplesPerSec=5.8798812802847955, CurrSamplesPerSec=5.354872344057848, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 84%|████████▍ | 4225/5000 [14:39:38<2:32:58, 11.84s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.6266666666666666e-06, 'epoch': 103.05}

 84%|████████▍ | 4225/5000 [14:39:38<2:32:58, 11.84s/it][2022-12-21 01:02:09,406] [INFO] [timer.py:197:stop] 0/8554, RunningAvgSamplesPerSec=5.879883356304084, CurrSamplesPerSec=5.3117999822937, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4226/5000 [14:39:49<2:29:38, 11.60s/it][2022-12-21 01:02:20,450] [INFO] [timer.py:197:stop] 0/8556, RunningAvgSamplesPerSec=5.879878871536576, CurrSamplesPerSec=5.30135519025362, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4227/5000 [14:40:00<2:27:12, 11.43s/it][2022-12-21 01:02:31,412] [INFO] [timer.py:197:stop] 0/8558, RunningAvgSamplesPerSec=5.879884578652951, CurrSamplesPerSec=5.352786507144388, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4228/5000 [14:40:11<2:25:10, 11.28s/it][2022-12-21 01:02:42,382] [INFO] [logging.py:68:log_dist] [Rank 0] step=4280, skipped=7, lr=[1.6177777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:02:42,383] [INFO] [timer.py:197:stop] 0/8560, RunningAvgSamplesPerSec=5.879888929517951, CurrSamplesPerSec=5.331704743132573, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4229/5000 [14:40:22<2:23:57, 11.20s/it][2022-12-21 01:02:53,417] [INFO] [timer.py:197:stop] 0/8562, RunningAvgSamplesPerSec=5.879885561740268, CurrSamplesPerSec=5.31531802610411, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4230/5000 [14:40:33<2:22:55, 11.14s/it][2022-12-21 01:03:04,371] [INFO] [timer.py:197:stop] 0/8564, RunningAvgSamplesPerSec=5.879892150280649, CurrSamplesPerSec=5.344075497114541, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4231/5000 [14:40:44<2:22:12, 11.10s/it][2022-12-21 01:03:15,384] [INFO] [timer.py:197:stop] 0/8566, RunningAvgSamplesPerSec=5.879891268952607, CurrSamplesPerSec=5.331303203956146, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4232/5000 [14:40:55<2:21:43, 11.07s/it][2022-12-21 01:03:26,361] [INFO] [timer.py:197:stop] 0/8568, RunningAvgSamplesPerSec=5.87989502399396, CurrSamplesPerSec=5.3672751987579, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4233/5000 [14:41:06<2:21:10, 11.04s/it][2022-12-21 01:03:37,344] [INFO] [timer.py:197:stop] 0/8570, RunningAvgSamplesPerSec=5.879898038157585, CurrSamplesPerSec=5.362581405479586, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4234/5000 [14:41:17<2:20:32, 11.01s/it][2022-12-21 01:03:48,272] [INFO] [timer.py:197:stop] 0/8572, RunningAvgSamplesPerSec=5.879907968693239, CurrSamplesPerSec=5.359425737115198, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4235/5000 [14:41:28<2:20:17, 11.00s/it][2022-12-21 01:03:59,298] [INFO] [timer.py:197:stop] 0/8574, RunningAvgSamplesPerSec=5.87990554838117, CurrSamplesPerSec=5.329536388003805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4236/5000 [14:41:39<2:20:12, 11.01s/it][2022-12-21 01:04:10,366] [INFO] [timer.py:197:stop] 0/8576, RunningAvgSamplesPerSec=5.879897826390295, CurrSamplesPerSec=5.293361506703441, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4237/5000 [14:41:50<2:20:01, 11.01s/it][2022-12-21 01:04:21,377] [INFO] [timer.py:197:stop] 0/8578, RunningAvgSamplesPerSec=5.879897284138656, CurrSamplesPerSec=5.294852702215552, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4238/5000 [14:42:01<2:19:57, 11.02s/it][2022-12-21 01:04:32,403] [INFO] [logging.py:68:log_dist] [Rank 0] step=4290, skipped=7, lr=[1.5955555555555558e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:04:32,404] [INFO] [timer.py:197:stop] 0/8580, RunningAvgSamplesPerSec=5.879894536827847, CurrSamplesPerSec=5.307809437636336, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4239/5000 [14:42:12<2:19:48, 11.02s/it][2022-12-21 01:04:43,454] [INFO] [timer.py:197:stop] 0/8582, RunningAvgSamplesPerSec=5.879889034920299, CurrSamplesPerSec=5.287650078809855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4240/5000 [14:42:23<2:19:44, 11.03s/it][2022-12-21 01:04:54,484] [INFO] [timer.py:197:stop] 0/8584, RunningAvgSamplesPerSec=5.8798862993144025, CurrSamplesPerSec=5.310039344884665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4241/5000 [14:42:34<2:19:22, 11.02s/it][2022-12-21 01:05:05,448] [INFO] [timer.py:197:stop] 0/8586, RunningAvgSamplesPerSec=5.879891725447864, CurrSamplesPerSec=5.329359262946111, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4242/5000 [14:42:45<2:19:09, 11.02s/it][2022-12-21 01:05:16,488] [INFO] [timer.py:197:stop] 0/8588, RunningAvgSamplesPerSec=5.879887849514769, CurrSamplesPerSec=5.302982470426443, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4243/5000 [14:42:56<2:18:58, 11.02s/it][2022-12-21 01:05:27,431] [INFO] [timer.py:197:stop] 0/8590, RunningAvgSamplesPerSec=5.87989594470536, CurrSamplesPerSec=5.366266824642769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4244/5000 [14:43:07<2:18:37, 11.00s/it][2022-12-21 01:05:38,439] [INFO] [timer.py:197:stop] 0/8592, RunningAvgSamplesPerSec=5.879895621243242, CurrSamplesPerSec=5.333646156856518, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4245/5000 [14:43:18<2:18:21, 11.00s/it][2022-12-21 01:05:49,451] [INFO] [timer.py:197:stop] 0/8594, RunningAvgSamplesPerSec=5.879894858601416, CurrSamplesPerSec=5.303500670002083, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4246/5000 [14:43:29<2:18:20, 11.01s/it][2022-12-21 01:06:00,501] [INFO] [timer.py:197:stop] 0/8596, RunningAvgSamplesPerSec=5.879889543442181, CurrSamplesPerSec=5.2945529756763205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4247/5000 [14:43:41<2:18:20, 11.02s/it][2022-12-21 01:06:11,555] [INFO] [timer.py:197:stop] 0/8598, RunningAvgSamplesPerSec=5.879883701203732, CurrSamplesPerSec=5.296595338869847, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4248/5000 [14:43:52<2:18:13, 11.03s/it][2022-12-21 01:06:22,529] [INFO] [logging.py:68:log_dist] [Rank 0] step=4300, skipped=7, lr=[1.5733333333333334e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:06:22,530] [INFO] [timer.py:197:stop] 0/8600, RunningAvgSamplesPerSec=5.87988772803385, CurrSamplesPerSec=5.354641192473486, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▍ | 4249/5000 [14:44:03<2:17:55, 11.02s/it][2022-12-21 01:06:33,607] [INFO] [timer.py:197:stop] 0/8602, RunningAvgSamplesPerSec=5.879878853463318, CurrSamplesPerSec=5.285681841067781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4250/5000 [14:44:14<2:17:57, 11.04s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.5711111111111113e-06, 'epoch': 103.65}

 85%|████████▌ | 4250/5000 [14:44:14<2:17:57, 11.04s/it][2022-12-21 01:06:44,675] [INFO] [timer.py:197:stop] 0/8604, RunningAvgSamplesPerSec=5.879871342136044, CurrSamplesPerSec=5.294750352752638, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4251/5000 [14:44:25<2:17:39, 11.03s/it][2022-12-21 01:06:55,660] [INFO] [timer.py:197:stop] 0/8606, RunningAvgSamplesPerSec=5.879874204267828, CurrSamplesPerSec=5.311861156949434, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4252/5000 [14:44:36<2:17:28, 11.03s/it][2022-12-21 01:07:06,675] [INFO] [timer.py:197:stop] 0/8608, RunningAvgSamplesPerSec=5.879873294541577, CurrSamplesPerSec=5.322786882625883, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4253/5000 [14:44:47<2:17:18, 11.03s/it][2022-12-21 01:07:17,774] [INFO] [timer.py:197:stop] 0/8610, RunningAvgSamplesPerSec=5.879861940846503, CurrSamplesPerSec=5.266259724618059, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4254/5000 [14:44:58<2:17:19, 11.04s/it][2022-12-21 01:07:28,764] [INFO] [timer.py:197:stop] 0/8612, RunningAvgSamplesPerSec=5.879864092982874, CurrSamplesPerSec=5.344479813874205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4255/5000 [14:45:09<2:16:59, 11.03s/it][2022-12-21 01:07:39,753] [INFO] [timer.py:197:stop] 0/8614, RunningAvgSamplesPerSec=5.879866575527532, CurrSamplesPerSec=5.360308230528786, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4256/5000 [14:45:20<2:16:32, 11.01s/it][2022-12-21 01:07:50,750] [INFO] [timer.py:197:stop] 0/8616, RunningAvgSamplesPerSec=5.879868026480436, CurrSamplesPerSec=5.330140013655588, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4257/5000 [14:45:31<2:16:14, 11.00s/it][2022-12-21 01:08:01,701] [INFO] [timer.py:197:stop] 0/8618, RunningAvgSamplesPerSec=5.87987494572535, CurrSamplesPerSec=5.352226403567573, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4258/5000 [14:45:42<2:16:01, 11.00s/it][2022-12-21 01:08:12,705] [INFO] [logging.py:68:log_dist] [Rank 0] step=4310, skipped=7, lr=[1.5511111111111113e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:08:12,706] [INFO] [timer.py:197:stop] 0/8620, RunningAvgSamplesPerSec=5.879875058091542, CurrSamplesPerSec=5.340283007164185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4259/5000 [14:45:53<2:15:41, 10.99s/it][2022-12-21 01:08:23,690] [INFO] [timer.py:197:stop] 0/8622, RunningAvgSamplesPerSec=5.879877931862296, CurrSamplesPerSec=5.321532457595625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4260/5000 [14:46:04<2:15:38, 11.00s/it][2022-12-21 01:08:34,684] [INFO] [timer.py:197:stop] 0/8624, RunningAvgSamplesPerSec=5.879879512683643, CurrSamplesPerSec=5.348227760895983, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4261/5000 [14:46:15<2:15:29, 11.00s/it][2022-12-21 01:08:45,736] [INFO] [timer.py:197:stop] 0/8626, RunningAvgSamplesPerSec=5.8798739388340735, CurrSamplesPerSec=5.30644688068591, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4262/5000 [14:46:26<2:15:29, 11.02s/it][2022-12-21 01:08:56,778] [INFO] [timer.py:197:stop] 0/8628, RunningAvgSamplesPerSec=5.879869601827597, CurrSamplesPerSec=5.3136442497315715, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4263/5000 [14:46:37<2:15:13, 11.01s/it][2022-12-21 01:09:07,801] [INFO] [timer.py:197:stop] 0/8630, RunningAvgSamplesPerSec=5.879867601905465, CurrSamplesPerSec=5.287326172565773, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4264/5000 [14:46:48<2:15:09, 11.02s/it][2022-12-21 01:09:17,865] [INFO] [timer.py:197:stop] 0/8632, RunningAvgSamplesPerSec=5.879984543501505, CurrSamplesPerSec=6.294213258795681, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 01:09:28,904] [INFO] [timer.py:197:stop] 0/8634, RunningAvgSamplesPerSec=5.879980464743204, CurrSamplesPerSec=5.32522736054657, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4265/5000 [14:47:04<2:33:56, 12.57s/it][2022-12-21 01:09:39,993] [INFO] [timer.py:197:stop] 0/8636, RunningAvgSamplesPerSec=5.879970258240734, CurrSamplesPerSec=5.281802967784431, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4266/5000 [14:47:15<2:28:18, 12.12s/it][2022-12-21 01:09:51,080] [INFO] [timer.py:197:stop] 0/8638, RunningAvgSamplesPerSec=5.879960384968637, CurrSamplesPerSec=5.295443064304069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4267/5000 [14:47:26<2:24:18, 11.81s/it][2022-12-21 01:10:02,080] [INFO] [logging.py:68:log_dist] [Rank 0] step=4320, skipped=7, lr=[1.5288888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:10:02,082] [INFO] [timer.py:197:stop] 0/8640, RunningAvgSamplesPerSec=5.879961180480427, CurrSamplesPerSec=5.3258177532406314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4268/5000 [14:47:37<2:21:08, 11.57s/it][2022-12-21 01:10:13,111] [INFO] [timer.py:197:stop] 0/8642, RunningAvgSamplesPerSec=5.879958571095166, CurrSamplesPerSec=5.337455117812139, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4269/5000 [14:47:48<2:18:58, 11.41s/it][2022-12-21 01:10:24,103] [INFO] [timer.py:197:stop] 0/8644, RunningAvgSamplesPerSec=5.8799605064346965, CurrSamplesPerSec=5.351816005741557, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4270/5000 [14:47:59<2:17:16, 11.28s/it][2022-12-21 01:10:35,113] [INFO] [timer.py:197:stop] 0/8646, RunningAvgSamplesPerSec=5.879960325666553, CurrSamplesPerSec=5.358436996278135, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4271/5000 [14:48:10<2:16:05, 11.20s/it][2022-12-21 01:10:46,138] [INFO] [timer.py:197:stop] 0/8648, RunningAvgSamplesPerSec=5.879958247187012, CurrSamplesPerSec=5.336512870549401, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4272/5000 [14:48:21<2:15:15, 11.15s/it][2022-12-21 01:10:57,118] [INFO] [timer.py:197:stop] 0/8650, RunningAvgSamplesPerSec=5.879961945320069, CurrSamplesPerSec=5.3632641197350805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4273/5000 [14:48:32<2:14:27, 11.10s/it][2022-12-21 01:11:08,186] [INFO] [timer.py:197:stop] 0/8652, RunningAvgSamplesPerSec=5.879954387250945, CurrSamplesPerSec=5.305256763635032, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 85%|████████▌ | 4274/5000 [14:48:43<2:14:10, 11.09s/it][2022-12-21 01:11:19,139] [INFO] [timer.py:197:stop] 0/8654, RunningAvgSamplesPerSec=5.879961193902788, CurrSamplesPerSec=5.369980301737027, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4275/5000 [14:48:54<2:13:29, 11.05s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.5133333333333334e-06, 'epoch': 104.27}

 86%|████████▌ | 4275/5000 [14:48:54<2:13:29, 11.05s/it][2022-12-21 01:11:30,152] [INFO] [timer.py:197:stop] 0/8656, RunningAvgSamplesPerSec=5.879960705135125, CurrSamplesPerSec=5.317077742918941, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4276/5000 [14:49:05<2:13:11, 11.04s/it][2022-12-21 01:11:41,167] [INFO] [timer.py:197:stop] 0/8658, RunningAvgSamplesPerSec=5.879959905847576, CurrSamplesPerSec=5.3146177904674685, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4277/5000 [14:49:16<2:12:55, 11.03s/it][2022-12-21 01:11:52,217] [INFO] [logging.py:68:log_dist] [Rank 0] step=4330, skipped=7, lr=[1.506666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:11:52,219] [INFO] [timer.py:197:stop] 0/8660, RunningAvgSamplesPerSec=5.879954485085444, CurrSamplesPerSec=5.303004470371737, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4278/5000 [14:49:27<2:12:48, 11.04s/it][2022-12-21 01:12:03,232] [INFO] [timer.py:197:stop] 0/8662, RunningAvgSamplesPerSec=5.879953796473493, CurrSamplesPerSec=5.3417982089827865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4279/5000 [14:49:38<2:12:32, 11.03s/it][2022-12-21 01:12:14,278] [INFO] [timer.py:197:stop] 0/8664, RunningAvgSamplesPerSec=5.879949135142992, CurrSamplesPerSec=5.292872003080344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4280/5000 [14:49:49<2:12:24, 11.03s/it][2022-12-21 01:12:25,285] [INFO] [timer.py:197:stop] 0/8666, RunningAvgSamplesPerSec=5.87994891615027, CurrSamplesPerSec=5.310129260830709, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4281/5000 [14:50:00<2:12:08, 11.03s/it][2022-12-21 01:12:36,312] [INFO] [timer.py:197:stop] 0/8668, RunningAvgSamplesPerSec=5.879946425276437, CurrSamplesPerSec=5.287395324779284, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4282/5000 [14:50:11<2:11:57, 11.03s/it][2022-12-21 01:12:47,373] [INFO] [timer.py:197:stop] 0/8670, RunningAvgSamplesPerSec=5.879939825754816, CurrSamplesPerSec=5.30556105894182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4283/5000 [14:50:22<2:11:53, 11.04s/it][2022-12-21 01:12:58,407] [INFO] [timer.py:197:stop] 0/8672, RunningAvgSamplesPerSec=5.879936613663335, CurrSamplesPerSec=5.297191107198777, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4284/5000 [14:50:34<2:11:41, 11.04s/it][2022-12-21 01:13:09,422] [INFO] [timer.py:197:stop] 0/8674, RunningAvgSamplesPerSec=5.879935544660305, CurrSamplesPerSec=5.33250482504106, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4285/5000 [14:50:45<2:11:26, 11.03s/it][2022-12-21 01:13:20,470] [INFO] [timer.py:197:stop] 0/8676, RunningAvgSamplesPerSec=5.879930340519865, CurrSamplesPerSec=5.322131738987805, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4286/5000 [14:50:56<2:11:19, 11.04s/it][2022-12-21 01:13:31,532] [INFO] [timer.py:197:stop] 0/8678, RunningAvgSamplesPerSec=5.879923446591428, CurrSamplesPerSec=5.302201485881988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4287/5000 [14:51:07<2:11:13, 11.04s/it][2022-12-21 01:13:42,478] [INFO] [logging.py:68:log_dist] [Rank 0] step=4340, skipped=7, lr=[1.4844444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:13:42,479] [INFO] [timer.py:197:stop] 0/8680, RunningAvgSamplesPerSec=5.879930888612194, CurrSamplesPerSec=5.355634516567394, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4288/5000 [14:51:18<2:10:42, 11.01s/it][2022-12-21 01:13:53,475] [INFO] [timer.py:197:stop] 0/8682, RunningAvgSamplesPerSec=5.8799323363587686, CurrSamplesPerSec=5.327342095534491, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4289/5000 [14:51:29<2:10:27, 11.01s/it][2022-12-21 01:14:04,465] [INFO] [timer.py:197:stop] 0/8684, RunningAvgSamplesPerSec=5.879934408790332, CurrSamplesPerSec=5.327639624766188, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4290/5000 [14:51:40<2:10:12, 11.00s/it][2022-12-21 01:14:15,460] [INFO] [timer.py:197:stop] 0/8686, RunningAvgSamplesPerSec=5.8799360204027735, CurrSamplesPerSec=5.308572968385374, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4291/5000 [14:51:51<2:09:59, 11.00s/it][2022-12-21 01:14:26,507] [INFO] [timer.py:197:stop] 0/8688, RunningAvgSamplesPerSec=5.879931048016617, CurrSamplesPerSec=5.317637044882575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4292/5000 [14:52:02<2:09:58, 11.01s/it][2022-12-21 01:14:37,666] [INFO] [timer.py:197:stop] 0/8690, RunningAvgSamplesPerSec=5.87991216257924, CurrSamplesPerSec=5.208657896838576, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4293/5000 [14:52:13<2:10:18, 11.06s/it][2022-12-21 01:14:48,704] [INFO] [timer.py:197:stop] 0/8692, RunningAvgSamplesPerSec=5.879908475357499, CurrSamplesPerSec=5.3174567071787235, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4294/5000 [14:52:24<2:10:02, 11.05s/it][2022-12-21 01:14:59,687] [INFO] [timer.py:197:stop] 0/8694, RunningAvgSamplesPerSec=5.879911392970317, CurrSamplesPerSec=5.319907058992474, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4295/5000 [14:52:35<2:09:37, 11.03s/it][2022-12-21 01:15:10,692] [INFO] [timer.py:197:stop] 0/8696, RunningAvgSamplesPerSec=5.8799118758698565, CurrSamplesPerSec=5.371700297593712, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4296/5000 [14:52:46<2:09:20, 11.02s/it][2022-12-21 01:15:21,705] [INFO] [timer.py:197:stop] 0/8698, RunningAvgSamplesPerSec=5.879911121242615, CurrSamplesPerSec=5.326974194893718, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4297/5000 [14:52:57<2:09:07, 11.02s/it][2022-12-21 01:15:32,692] [INFO] [logging.py:68:log_dist] [Rank 0] step=4350, skipped=7, lr=[1.4622222222222224e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:15:32,694] [INFO] [timer.py:197:stop] 0/8700, RunningAvgSamplesPerSec=5.879913337321643, CurrSamplesPerSec=5.348110977532254, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4298/5000 [14:53:08<2:08:49, 11.01s/it][2022-12-21 01:15:43,660] [INFO] [timer.py:197:stop] 0/8702, RunningAvgSamplesPerSec=5.8799184575484045, CurrSamplesPerSec=5.343968895360794, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4299/5000 [14:53:19<2:08:29, 11.00s/it][2022-12-21 01:15:54,654] [INFO] [timer.py:197:stop] 0/8704, RunningAvgSamplesPerSec=5.879920080703591, CurrSamplesPerSec=5.338558224402548, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4300/5000 [14:53:30<2:08:17, 11.00s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.457777777777778e-06, 'epoch': 104.87}

 86%|████████▌ | 4300/5000 [14:53:30<2:08:17, 11.00s/it][2022-12-21 01:16:05,646] [INFO] [timer.py:197:stop] 0/8706, RunningAvgSamplesPerSec=5.879922194563794, CurrSamplesPerSec=5.361453143773343, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4301/5000 [14:53:41<2:08:05, 11.00s/it][2022-12-21 01:16:16,609] [INFO] [timer.py:197:stop] 0/8708, RunningAvgSamplesPerSec=5.87992776655437, CurrSamplesPerSec=5.348025950517929, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4302/5000 [14:53:52<2:07:47, 10.99s/it][2022-12-21 01:16:27,604] [INFO] [timer.py:197:stop] 0/8710, RunningAvgSamplesPerSec=5.879929449259168, CurrSamplesPerSec=5.3373757355021905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4303/5000 [14:54:03<2:07:38, 10.99s/it][2022-12-21 01:16:38,618] [INFO] [timer.py:197:stop] 0/8712, RunningAvgSamplesPerSec=5.879928735516116, CurrSamplesPerSec=5.325228839534356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4304/5000 [14:54:14<2:07:33, 11.00s/it][2022-12-21 01:16:49,612] [INFO] [timer.py:197:stop] 0/8714, RunningAvgSamplesPerSec=5.879930430650143, CurrSamplesPerSec=5.347734875997547, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4305/5000 [14:54:25<2:07:21, 11.00s/it][2022-12-21 01:16:59,729] [INFO] [timer.py:197:stop] 0/8716, RunningAvgSamplesPerSec=5.880039613304419, CurrSamplesPerSec=5.307166579484311, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4306/5000 [14:54:40<2:21:07, 12.20s/it][2022-12-21 01:17:10,667] [INFO] [timer.py:197:stop] 0/8718, RunningAvgSamplesPerSec=5.880048107645109, CurrSamplesPerSec=5.3549135774452425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4307/5000 [14:54:51<2:16:26, 11.81s/it][2022-12-21 01:17:21,601] [INFO] [logging.py:68:log_dist] [Rank 0] step=4360, skipped=7, lr=[1.44e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:17:21,603] [INFO] [timer.py:197:stop] 0/8720, RunningAvgSamplesPerSec=5.880056886831902, CurrSamplesPerSec=5.327961932730861, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4308/5000 [14:55:02<2:13:22, 11.56s/it][2022-12-21 01:17:32,540] [INFO] [timer.py:197:stop] 0/8722, RunningAvgSamplesPerSec=5.880065445682634, CurrSamplesPerSec=5.371005332199622, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4309/5000 [14:55:13<2:11:02, 11.38s/it][2022-12-21 01:17:43,509] [INFO] [timer.py:197:stop] 0/8724, RunningAvgSamplesPerSec=5.880070172640005, CurrSamplesPerSec=5.348568976547431, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4310/5000 [14:55:24<2:09:25, 11.25s/it][2022-12-21 01:17:54,469] [INFO] [timer.py:197:stop] 0/8726, RunningAvgSamplesPerSec=5.8800761015477, CurrSamplesPerSec=5.3544880281562595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4311/5000 [14:55:34<2:08:16, 11.17s/it][2022-12-21 01:18:05,422] [INFO] [timer.py:197:stop] 0/8728, RunningAvgSamplesPerSec=5.8800827255545425, CurrSamplesPerSec=5.37171857161256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▌ | 4312/5000 [14:55:45<2:07:22, 11.11s/it][2022-12-21 01:18:16,450] [INFO] [timer.py:197:stop] 0/8730, RunningAvgSamplesPerSec=5.880080170066392, CurrSamplesPerSec=5.313044143981737, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4313/5000 [14:55:57<2:07:00, 11.09s/it][2022-12-21 01:18:27,491] [INFO] [timer.py:197:stop] 0/8732, RunningAvgSamplesPerSec=5.880075799377713, CurrSamplesPerSec=5.32502052163232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4314/5000 [14:56:08<2:06:31, 11.07s/it][2022-12-21 01:18:38,465] [INFO] [timer.py:197:stop] 0/8734, RunningAvgSamplesPerSec=5.880079823100469, CurrSamplesPerSec=5.354751424935762, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4315/5000 [14:56:19<2:06:09, 11.05s/it][2022-12-21 01:18:49,532] [INFO] [timer.py:197:stop] 0/8736, RunningAvgSamplesPerSec=5.8800723236712535, CurrSamplesPerSec=5.304174292221401, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4316/5000 [14:56:30<2:05:56, 11.05s/it][2022-12-21 01:19:00,545] [INFO] [timer.py:197:stop] 0/8738, RunningAvgSamplesPerSec=5.880071643207836, CurrSamplesPerSec=5.331397653406494, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4317/5000 [14:56:41<2:05:38, 11.04s/it][2022-12-21 01:19:11,558] [INFO] [logging.py:68:log_dist] [Rank 0] step=4370, skipped=7, lr=[1.417777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:19:11,560] [INFO] [timer.py:197:stop] 0/8740, RunningAvgSamplesPerSec=5.8800707027087755, CurrSamplesPerSec=5.32888190923151, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4318/5000 [14:56:52<2:05:23, 11.03s/it][2022-12-21 01:19:22,564] [INFO] [timer.py:197:stop] 0/8742, RunningAvgSamplesPerSec=5.880070989803587, CurrSamplesPerSec=5.342152638177304, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4319/5000 [14:57:03<2:05:00, 11.01s/it][2022-12-21 01:19:33,602] [INFO] [timer.py:197:stop] 0/8744, RunningAvgSamplesPerSec=5.880067259631201, CurrSamplesPerSec=5.286577486558131, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4320/5000 [14:57:14<2:05:01, 11.03s/it][2022-12-21 01:19:44,535] [INFO] [timer.py:197:stop] 0/8746, RunningAvgSamplesPerSec=5.8800763515501755, CurrSamplesPerSec=5.407855504425435, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4321/5000 [14:57:25<2:04:31, 11.00s/it][2022-12-21 01:19:55,512] [INFO] [timer.py:197:stop] 0/8748, RunningAvgSamplesPerSec=5.8800800785827, CurrSamplesPerSec=5.375387931997748, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4322/5000 [14:57:36<2:04:08, 10.99s/it][2022-12-21 01:20:06,545] [INFO] [timer.py:197:stop] 0/8750, RunningAvgSamplesPerSec=5.880076926461079, CurrSamplesPerSec=5.296940868177069, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4323/5000 [14:57:47<2:04:11, 11.01s/it][2022-12-21 01:20:17,530] [INFO] [timer.py:197:stop] 0/8752, RunningAvgSamplesPerSec=5.880080131411976, CurrSamplesPerSec=5.360824205325838, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4324/5000 [14:57:58<2:03:54, 11.00s/it][2022-12-21 01:20:28,548] [INFO] [timer.py:197:stop] 0/8754, RunningAvgSamplesPerSec=5.880078761680582, CurrSamplesPerSec=5.320887747784542, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 86%|████████▋ | 4325/5000 [14:58:09<2:03:52, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.4022222222222223e-06, 'epoch': 105.48}

 86%|████████▋ | 4325/5000 [14:58:09<2:03:52, 11.01s/it][2022-12-21 01:20:39,582] [INFO] [timer.py:197:stop] 0/8756, RunningAvgSamplesPerSec=5.880075557846176, CurrSamplesPerSec=5.331276521533824, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4326/5000 [14:58:20<2:03:46, 11.02s/it][2022-12-21 01:20:50,656] [INFO] [timer.py:197:stop] 0/8758, RunningAvgSamplesPerSec=5.8800673407923165, CurrSamplesPerSec=5.295084778614407, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4327/5000 [14:58:31<2:03:45, 11.03s/it][2022-12-21 01:21:01,705] [INFO] [logging.py:68:log_dist] [Rank 0] step=4380, skipped=7, lr=[1.3955555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:21:01,706] [INFO] [timer.py:197:stop] 0/8760, RunningAvgSamplesPerSec=5.8800618286276665, CurrSamplesPerSec=5.309685172906127, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4328/5000 [14:58:42<2:03:25, 11.02s/it][2022-12-21 01:21:12,638] [INFO] [timer.py:197:stop] 0/8762, RunningAvgSamplesPerSec=5.880070949862246, CurrSamplesPerSec=5.3611417611812575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4329/5000 [14:58:53<2:03:02, 11.00s/it][2022-12-21 01:21:23,650] [INFO] [timer.py:197:stop] 0/8764, RunningAvgSamplesPerSec=5.880070233070502, CurrSamplesPerSec=5.315985810648288, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4330/5000 [14:59:04<2:02:51, 11.00s/it][2022-12-21 01:21:34,651] [INFO] [timer.py:197:stop] 0/8766, RunningAvgSamplesPerSec=5.880070864567155, CurrSamplesPerSec=5.316793818439232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4331/5000 [14:59:15<2:02:39, 11.00s/it][2022-12-21 01:21:45,667] [INFO] [timer.py:197:stop] 0/8768, RunningAvgSamplesPerSec=5.880069792628753, CurrSamplesPerSec=5.300094311052882, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4332/5000 [14:59:26<2:02:41, 11.02s/it][2022-12-21 01:21:56,709] [INFO] [timer.py:197:stop] 0/8770, RunningAvgSamplesPerSec=5.880065573477631, CurrSamplesPerSec=5.321472958851297, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4333/5000 [14:59:37<2:02:35, 11.03s/it][2022-12-21 01:22:07,683] [INFO] [timer.py:197:stop] 0/8772, RunningAvgSamplesPerSec=5.880069557703866, CurrSamplesPerSec=5.384811429313304, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4334/5000 [14:59:48<2:02:09, 11.01s/it][2022-12-21 01:22:18,717] [INFO] [timer.py:197:stop] 0/8774, RunningAvgSamplesPerSec=5.880066126120435, CurrSamplesPerSec=5.3133823572609025, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4335/5000 [14:59:59<2:02:00, 11.01s/it][2022-12-21 01:22:29,695] [INFO] [timer.py:197:stop] 0/8776, RunningAvgSamplesPerSec=5.880069581508801, CurrSamplesPerSec=5.34352232067107, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4336/5000 [15:00:10<2:01:34, 10.99s/it][2022-12-21 01:22:40,672] [INFO] [timer.py:197:stop] 0/8778, RunningAvgSamplesPerSec=5.880073123357254, CurrSamplesPerSec=5.30411874440192, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4337/5000 [15:00:21<2:01:27, 10.99s/it][2022-12-21 01:22:51,689] [INFO] [logging.py:68:log_dist] [Rank 0] step=4390, skipped=7, lr=[1.3733333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:22:51,691] [INFO] [timer.py:197:stop] 0/8780, RunningAvgSamplesPerSec=5.880071600076886, CurrSamplesPerSec=5.293554619388873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4338/5000 [15:00:32<2:01:16, 10.99s/it][2022-12-21 01:23:02,648] [INFO] [timer.py:197:stop] 0/8782, RunningAvgSamplesPerSec=5.880077681704295, CurrSamplesPerSec=5.323924902590398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4339/5000 [15:00:43<2:01:08, 11.00s/it][2022-12-21 01:23:13,684] [INFO] [timer.py:197:stop] 0/8784, RunningAvgSamplesPerSec=5.880074089104433, CurrSamplesPerSec=5.29987958384461, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4340/5000 [15:00:54<2:01:00, 11.00s/it][2022-12-21 01:23:24,682] [INFO] [timer.py:197:stop] 0/8786, RunningAvgSamplesPerSec=5.880075109592547, CurrSamplesPerSec=5.312404222621448, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4341/5000 [15:01:05<2:00:53, 11.01s/it][2022-12-21 01:23:35,706] [INFO] [timer.py:197:stop] 0/8788, RunningAvgSamplesPerSec=5.880072963403767, CurrSamplesPerSec=5.309350580509899, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4342/5000 [15:01:16<2:00:44, 11.01s/it][2022-12-21 01:23:46,726] [INFO] [timer.py:197:stop] 0/8790, RunningAvgSamplesPerSec=5.880071248132091, CurrSamplesPerSec=5.307777952219698, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4343/5000 [15:01:27<2:00:41, 11.02s/it][2022-12-21 01:23:57,722] [INFO] [timer.py:197:stop] 0/8792, RunningAvgSamplesPerSec=5.880072832429559, CurrSamplesPerSec=5.35689951360828, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4344/5000 [15:01:38<2:00:24, 11.01s/it][2022-12-21 01:24:08,783] [INFO] [timer.py:197:stop] 0/8794, RunningAvgSamplesPerSec=5.880066115349277, CurrSamplesPerSec=5.294030885051754, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4345/5000 [15:01:49<2:00:19, 11.02s/it][2022-12-21 01:24:19,765] [INFO] [timer.py:197:stop] 0/8796, RunningAvgSamplesPerSec=5.88006928307179, CurrSamplesPerSec=5.346399237549614, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4346/5000 [15:02:00<2:00:00, 11.01s/it][2022-12-21 01:24:29,864] [INFO] [timer.py:197:stop] 0/8798, RunningAvgSamplesPerSec=5.880179758382183, CurrSamplesPerSec=6.260061498146413, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 01:24:40,866] [INFO] [logging.py:68:log_dist] [Rank 0] step=4400, skipped=7, lr=[1.351111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:24:40,868] [INFO] [timer.py:197:stop] 0/8800, RunningAvgSamplesPerSec=5.880179736182276, CurrSamplesPerSec=5.338708355172277, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4347/5000 [15:02:16<2:16:40, 12.56s/it][2022-12-21 01:24:51,904] [INFO] [timer.py:197:stop] 0/8802, RunningAvgSamplesPerSec=5.88017612778753, CurrSamplesPerSec=5.313580930390437, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4348/5000 [15:02:27<2:11:29, 12.10s/it][2022-12-21 01:25:02,913] [INFO] [timer.py:197:stop] 0/8804, RunningAvgSamplesPerSec=5.880175885743819, CurrSamplesPerSec=5.3382061828149014, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4349/5000 [15:02:38<2:07:44, 11.77s/it][2022-12-21 01:25:13,936] [INFO] [timer.py:197:stop] 0/8806, RunningAvgSamplesPerSec=5.880173870558577, CurrSamplesPerSec=5.310832096676982, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4350/5000 [15:02:49<2:05:06, 11.55s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.3444444444444446e-06, 'epoch': 106.1}

 87%|████████▋ | 4350/5000 [15:02:49<2:05:06, 11.55s/it][2022-12-21 01:25:24,927] [INFO] [timer.py:197:stop] 0/8808, RunningAvgSamplesPerSec=5.880176013053391, CurrSamplesPerSec=5.3212305468701615, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4351/5000 [15:03:00<2:03:06, 11.38s/it][2022-12-21 01:25:35,986] [INFO] [timer.py:197:stop] 0/8810, RunningAvgSamplesPerSec=5.880169659993011, CurrSamplesPerSec=5.323630110837856, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4352/5000 [15:03:11<2:01:52, 11.28s/it][2022-12-21 01:25:46,999] [INFO] [timer.py:197:stop] 0/8812, RunningAvgSamplesPerSec=5.8801687924745965, CurrSamplesPerSec=5.322088054410409, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4353/5000 [15:03:22<2:00:48, 11.20s/it][2022-12-21 01:25:58,035] [INFO] [timer.py:197:stop] 0/8814, RunningAvgSamplesPerSec=5.880165118784875, CurrSamplesPerSec=5.332918199429803, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4354/5000 [15:03:33<2:00:04, 11.15s/it][2022-12-21 01:26:09,084] [INFO] [timer.py:197:stop] 0/8816, RunningAvgSamplesPerSec=5.880159984385435, CurrSamplesPerSec=5.290290091909606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4355/5000 [15:03:44<1:59:33, 11.12s/it][2022-12-21 01:26:20,047] [INFO] [timer.py:197:stop] 0/8818, RunningAvgSamplesPerSec=5.880165469730546, CurrSamplesPerSec=5.343558060918867, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4356/5000 [15:03:55<1:58:51, 11.07s/it][2022-12-21 01:26:31,050] [INFO] [logging.py:68:log_dist] [Rank 0] step=4410, skipped=7, lr=[1.3288888888888891e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:26:31,051] [INFO] [timer.py:197:stop] 0/8820, RunningAvgSamplesPerSec=5.880165693390253, CurrSamplesPerSec=5.342179642209602, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4357/5000 [15:04:06<1:58:27, 11.05s/it][2022-12-21 01:26:42,056] [INFO] [timer.py:197:stop] 0/8822, RunningAvgSamplesPerSec=5.880165953633696, CurrSamplesPerSec=5.324351943954912, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4358/5000 [15:04:17<1:58:06, 11.04s/it][2022-12-21 01:26:53,105] [INFO] [timer.py:197:stop] 0/8824, RunningAvgSamplesPerSec=5.880160880174749, CurrSamplesPerSec=5.304343667849356, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4359/5000 [15:04:28<1:57:57, 11.04s/it][2022-12-21 01:27:04,138] [INFO] [timer.py:197:stop] 0/8826, RunningAvgSamplesPerSec=5.880157562480806, CurrSamplesPerSec=5.3034748938328855, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4360/5000 [15:04:39<1:57:45, 11.04s/it][2022-12-21 01:27:15,137] [INFO] [timer.py:197:stop] 0/8828, RunningAvgSamplesPerSec=5.880158614595609, CurrSamplesPerSec=5.3167371635483684, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4361/5000 [15:04:50<1:57:26, 11.03s/it][2022-12-21 01:27:26,217] [INFO] [timer.py:197:stop] 0/8830, RunningAvgSamplesPerSec=5.880149787502633, CurrSamplesPerSec=5.3169802194450195, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4362/5000 [15:05:01<1:57:25, 11.04s/it][2022-12-21 01:27:37,189] [INFO] [timer.py:197:stop] 0/8832, RunningAvgSamplesPerSec=5.880154016741563, CurrSamplesPerSec=5.3427170154286205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4363/5000 [15:05:12<1:57:00, 11.02s/it][2022-12-21 01:27:48,237] [INFO] [timer.py:197:stop] 0/8834, RunningAvgSamplesPerSec=5.880148927978703, CurrSamplesPerSec=5.325301522231983, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4364/5000 [15:05:23<1:56:54, 11.03s/it][2022-12-21 01:27:59,248] [INFO] [timer.py:197:stop] 0/8836, RunningAvgSamplesPerSec=5.880148352016119, CurrSamplesPerSec=5.304371968049767, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4365/5000 [15:05:34<1:56:40, 11.02s/it][2022-12-21 01:28:10,282] [INFO] [timer.py:197:stop] 0/8838, RunningAvgSamplesPerSec=5.8801450143887655, CurrSamplesPerSec=5.3021426280758535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4366/5000 [15:05:45<1:56:31, 11.03s/it][2022-12-21 01:28:21,234] [INFO] [logging.py:68:log_dist] [Rank 0] step=4420, skipped=7, lr=[1.3066666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:28:21,236] [INFO] [timer.py:197:stop] 0/8840, RunningAvgSamplesPerSec=5.880151304658582, CurrSamplesPerSec=5.371329254410301, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4367/5000 [15:05:56<1:56:06, 11.01s/it][2022-12-21 01:28:32,247] [INFO] [timer.py:197:stop] 0/8842, RunningAvgSamplesPerSec=5.880150740759115, CurrSamplesPerSec=5.354338717403136, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4368/5000 [15:06:07<1:55:56, 11.01s/it][2022-12-21 01:28:43,215] [INFO] [timer.py:197:stop] 0/8844, RunningAvgSamplesPerSec=5.880155449493827, CurrSamplesPerSec=5.39016434576847, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4369/5000 [15:06:18<1:55:37, 11.00s/it][2022-12-21 01:28:54,270] [INFO] [timer.py:197:stop] 0/8846, RunningAvgSamplesPerSec=5.88014958794691, CurrSamplesPerSec=5.307182948082875, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4370/5000 [15:06:29<1:55:38, 11.01s/it][2022-12-21 01:29:05,339] [INFO] [timer.py:197:stop] 0/8848, RunningAvgSamplesPerSec=5.8801419687046685, CurrSamplesPerSec=5.292720056191641, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4371/5000 [15:06:40<1:55:37, 11.03s/it][2022-12-21 01:29:16,327] [INFO] [timer.py:197:stop] 0/8850, RunningAvgSamplesPerSec=5.8801443059974146, CurrSamplesPerSec=5.336876360311179, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4372/5000 [15:06:51<1:55:18, 11.02s/it][2022-12-21 01:29:27,354] [INFO] [timer.py:197:stop] 0/8852, RunningAvgSamplesPerSec=5.8801418618885, CurrSamplesPerSec=5.338357992401362, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4373/5000 [15:07:02<1:55:09, 11.02s/it][2022-12-21 01:29:38,347] [INFO] [timer.py:197:stop] 0/8854, RunningAvgSamplesPerSec=5.8801436356633605, CurrSamplesPerSec=5.364568745841959, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 87%|████████▋ | 4374/5000 [15:07:13<1:54:53, 11.01s/it][2022-12-21 01:29:49,329] [INFO] [timer.py:197:stop] 0/8856, RunningAvgSamplesPerSec=5.880146807756508, CurrSamplesPerSec=5.337991753215954, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4375/5000 [15:07:24<1:54:36, 11.00s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.288888888888889e-06, 'epoch': 106.7}

 88%|████████▊ | 4375/5000 [15:07:24<1:54:36, 11.00s/it][2022-12-21 01:30:00,327] [INFO] [timer.py:197:stop] 0/8858, RunningAvgSamplesPerSec=5.880148016976436, CurrSamplesPerSec=5.35105193287398, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4376/5000 [15:07:35<1:54:24, 11.00s/it][2022-12-21 01:30:11,374] [INFO] [logging.py:68:log_dist] [Rank 0] step=4430, skipped=7, lr=[1.2844444444444445e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:30:11,375] [INFO] [timer.py:197:stop] 0/8860, RunningAvgSamplesPerSec=5.8801429269202545, CurrSamplesPerSec=5.313219765527942, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4377/5000 [15:07:46<1:54:22, 11.02s/it][2022-12-21 01:30:22,386] [INFO] [timer.py:197:stop] 0/8862, RunningAvgSamplesPerSec=5.880142474086831, CurrSamplesPerSec=5.297064834929426, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4378/5000 [15:07:57<1:54:10, 11.01s/it][2022-12-21 01:30:33,367] [INFO] [timer.py:197:stop] 0/8864, RunningAvgSamplesPerSec=5.880145551747417, CurrSamplesPerSec=5.315032395239184, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4379/5000 [15:08:08<1:53:53, 11.00s/it][2022-12-21 01:30:44,430] [INFO] [timer.py:197:stop] 0/8866, RunningAvgSamplesPerSec=5.880138656167615, CurrSamplesPerSec=5.303272255766982, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4380/5000 [15:08:20<1:53:53, 11.02s/it][2022-12-21 01:30:55,377] [INFO] [timer.py:197:stop] 0/8868, RunningAvgSamplesPerSec=5.880145835242804, CurrSamplesPerSec=5.346840755184924, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4381/5000 [15:08:30<1:53:28, 11.00s/it][2022-12-21 01:31:06,379] [INFO] [timer.py:197:stop] 0/8870, RunningAvgSamplesPerSec=5.880146494094314, CurrSamplesPerSec=5.320184567786748, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4382/5000 [15:08:41<1:53:18, 11.00s/it][2022-12-21 01:31:17,377] [INFO] [timer.py:197:stop] 0/8872, RunningAvgSamplesPerSec=5.880147581963907, CurrSamplesPerSec=5.33573216095245, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4383/5000 [15:08:52<1:53:06, 11.00s/it][2022-12-21 01:31:28,342] [INFO] [timer.py:197:stop] 0/8874, RunningAvgSamplesPerSec=5.880152690939088, CurrSamplesPerSec=5.335794100218102, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4384/5000 [15:09:03<1:52:49, 10.99s/it][2022-12-21 01:31:39,351] [INFO] [timer.py:197:stop] 0/8876, RunningAvgSamplesPerSec=5.880152326146666, CurrSamplesPerSec=5.343331501367405, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4385/5000 [15:09:14<1:52:42, 11.00s/it][2022-12-21 01:31:50,355] [INFO] [timer.py:197:stop] 0/8878, RunningAvgSamplesPerSec=5.880152626853685, CurrSamplesPerSec=5.329364764861524, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4386/5000 [15:09:25<1:52:32, 11.00s/it][2022-12-21 01:32:01,315] [INFO] [logging.py:68:log_dist] [Rank 0] step=4440, skipped=7, lr=[1.2622222222222224e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:32:01,317] [INFO] [timer.py:197:stop] 0/8880, RunningAvgSamplesPerSec=5.880158026351529, CurrSamplesPerSec=5.354494222916769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4387/5000 [15:09:36<1:52:15, 10.99s/it][2022-12-21 01:32:11,388] [INFO] [timer.py:197:stop] 0/8882, RunningAvgSamplesPerSec=5.880270546481734, CurrSamplesPerSec=5.348745675590254, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4388/5000 [15:09:51<2:04:22, 12.19s/it][2022-12-21 01:32:22,412] [INFO] [timer.py:197:stop] 0/8884, RunningAvgSamplesPerSec=5.8802683284069275, CurrSamplesPerSec=5.312713544260577, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4389/5000 [15:10:02<2:00:32, 11.84s/it][2022-12-21 01:32:33,409] [INFO] [timer.py:197:stop] 0/8886, RunningAvgSamplesPerSec=5.880269280783198, CurrSamplesPerSec=5.320362559790182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4390/5000 [15:10:13<1:57:53, 11.60s/it][2022-12-21 01:32:44,407] [INFO] [timer.py:197:stop] 0/8888, RunningAvgSamplesPerSec=5.880270301123228, CurrSamplesPerSec=5.351869782840479, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4391/5000 [15:10:24<1:55:46, 11.41s/it][2022-12-21 01:32:55,419] [INFO] [timer.py:197:stop] 0/8890, RunningAvgSamplesPerSec=5.880269597700981, CurrSamplesPerSec=5.31078334391129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4392/5000 [15:10:35<1:54:29, 11.30s/it][2022-12-21 01:33:06,447] [INFO] [timer.py:197:stop] 0/8892, RunningAvgSamplesPerSec=5.88026699943028, CurrSamplesPerSec=5.325827263141906, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4393/5000 [15:10:47<1:53:33, 11.22s/it][2022-12-21 01:33:17,496] [INFO] [timer.py:197:stop] 0/8894, RunningAvgSamplesPerSec=5.880261975719917, CurrSamplesPerSec=5.330734036868159, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4394/5000 [15:10:58<1:52:46, 11.17s/it][2022-12-21 01:33:28,470] [INFO] [timer.py:197:stop] 0/8896, RunningAvgSamplesPerSec=5.880265714245118, CurrSamplesPerSec=5.377370558305674, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4395/5000 [15:11:08<1:51:53, 11.10s/it][2022-12-21 01:33:39,415] [INFO] [timer.py:197:stop] 0/8898, RunningAvgSamplesPerSec=5.880273135573905, CurrSamplesPerSec=5.36779960003577, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4396/5000 [15:11:19<1:51:23, 11.07s/it][2022-12-21 01:33:50,448] [INFO] [logging.py:68:log_dist] [Rank 0] step=4450, skipped=7, lr=[1.2400000000000002e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:33:50,450] [INFO] [timer.py:197:stop] 0/8900, RunningAvgSamplesPerSec=5.880269815857711, CurrSamplesPerSec=5.332768606013655, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4397/5000 [15:11:30<1:50:55, 11.04s/it][2022-12-21 01:34:01,412] [INFO] [timer.py:197:stop] 0/8902, RunningAvgSamplesPerSec=5.880275096372063, CurrSamplesPerSec=5.341168559380637, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4398/5000 [15:11:41<1:50:33, 11.02s/it][2022-12-21 01:34:12,405] [INFO] [timer.py:197:stop] 0/8904, RunningAvgSamplesPerSec=5.88027673682472, CurrSamplesPerSec=5.326456471395012, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4399/5000 [15:11:52<1:50:24, 11.02s/it][2022-12-21 01:34:23,396] [INFO] [timer.py:197:stop] 0/8906, RunningAvgSamplesPerSec=5.8802786888204555, CurrSamplesPerSec=5.359688763810507, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4400/5000 [15:12:03<1:50:05, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.2333333333333335e-06, 'epoch': 107.31}

 88%|████████▊ | 4400/5000 [15:12:03<1:50:05, 11.01s/it][2022-12-21 01:34:34,393] [INFO] [timer.py:197:stop] 0/8908, RunningAvgSamplesPerSec=5.880281069535819, CurrSamplesPerSec=5.347503061857896, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4401/5000 [15:12:14<1:49:47, 11.00s/it][2022-12-21 01:34:45,343] [INFO] [timer.py:197:stop] 0/8910, RunningAvgSamplesPerSec=5.88028800914735, CurrSamplesPerSec=5.361345847612888, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4402/5000 [15:12:25<1:49:33, 10.99s/it][2022-12-21 01:34:56,351] [INFO] [timer.py:197:stop] 0/8912, RunningAvgSamplesPerSec=5.880287755410082, CurrSamplesPerSec=5.3359336809580835, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4403/5000 [15:12:36<1:49:25, 11.00s/it][2022-12-21 01:35:07,384] [INFO] [timer.py:197:stop] 0/8914, RunningAvgSamplesPerSec=5.8802845052551795, CurrSamplesPerSec=5.315763898936041, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4404/5000 [15:12:47<1:49:18, 11.00s/it][2022-12-21 01:35:18,480] [INFO] [timer.py:197:stop] 0/8916, RunningAvgSamplesPerSec=5.880273760100813, CurrSamplesPerSec=5.251975765528925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4405/5000 [15:12:59<1:49:25, 11.03s/it][2022-12-21 01:35:29,469] [INFO] [timer.py:197:stop] 0/8918, RunningAvgSamplesPerSec=5.8802757683286755, CurrSamplesPerSec=5.352019169341559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4406/5000 [15:13:10<1:49:04, 11.02s/it][2022-12-21 01:35:40,446] [INFO] [logging.py:68:log_dist] [Rank 0] step=4460, skipped=7, lr=[1.2177777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:35:40,448] [INFO] [timer.py:197:stop] 0/8920, RunningAvgSamplesPerSec=5.880279087810776, CurrSamplesPerSec=5.35352908121962, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4407/5000 [15:13:20<1:48:39, 10.99s/it][2022-12-21 01:35:51,430] [INFO] [timer.py:197:stop] 0/8922, RunningAvgSamplesPerSec=5.880281937752482, CurrSamplesPerSec=5.314821086044379, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4408/5000 [15:13:31<1:48:37, 11.01s/it][2022-12-21 01:36:02,449] [INFO] [timer.py:197:stop] 0/8924, RunningAvgSamplesPerSec=5.880280437243664, CurrSamplesPerSec=5.336977161629642, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4409/5000 [15:13:42<1:48:24, 11.01s/it][2022-12-21 01:36:13,455] [INFO] [timer.py:197:stop] 0/8926, RunningAvgSamplesPerSec=5.880280549404935, CurrSamplesPerSec=5.331563689251887, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4410/5000 [15:13:54<1:48:18, 11.01s/it][2022-12-21 01:36:24,542] [INFO] [timer.py:197:stop] 0/8928, RunningAvgSamplesPerSec=5.880270746589671, CurrSamplesPerSec=5.2856108602987275, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4411/5000 [15:14:05<1:48:12, 11.02s/it][2022-12-21 01:36:35,510] [INFO] [timer.py:197:stop] 0/8930, RunningAvgSamplesPerSec=5.880275401193019, CurrSamplesPerSec=5.351605389176193, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4412/5000 [15:14:16<1:47:55, 11.01s/it][2022-12-21 01:36:46,491] [INFO] [timer.py:197:stop] 0/8932, RunningAvgSamplesPerSec=5.880278509504864, CurrSamplesPerSec=5.360371169852717, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4413/5000 [15:14:27<1:47:40, 11.01s/it][2022-12-21 01:36:57,476] [INFO] [timer.py:197:stop] 0/8934, RunningAvgSamplesPerSec=5.880281172912061, CurrSamplesPerSec=5.361295734709873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4414/5000 [15:14:38<1:47:24, 11.00s/it][2022-12-21 01:37:08,517] [INFO] [timer.py:197:stop] 0/8936, RunningAvgSamplesPerSec=5.880277071016022, CurrSamplesPerSec=5.306270238108181, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4415/5000 [15:14:49<1:47:26, 11.02s/it][2022-12-21 01:37:19,550] [INFO] [timer.py:197:stop] 0/8938, RunningAvgSamplesPerSec=5.880274093537571, CurrSamplesPerSec=5.339054091516246, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4416/5000 [15:15:00<1:47:15, 11.02s/it][2022-12-21 01:37:30,585] [INFO] [logging.py:68:log_dist] [Rank 0] step=4470, skipped=7, lr=[1.1955555555555556e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:37:30,586] [INFO] [timer.py:197:stop] 0/8940, RunningAvgSamplesPerSec=5.880270548479393, CurrSamplesPerSec=5.324862286978595, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4417/5000 [15:15:11<1:47:09, 11.03s/it][2022-12-21 01:37:41,627] [INFO] [timer.py:197:stop] 0/8942, RunningAvgSamplesPerSec=5.880268637227993, CurrSamplesPerSec=5.330472785831584, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4418/5000 [15:15:22<1:46:58, 11.03s/it][2022-12-21 01:37:52,677] [INFO] [timer.py:197:stop] 0/8944, RunningAvgSamplesPerSec=5.880263390473467, CurrSamplesPerSec=5.312484335782559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4419/5000 [15:15:33<1:46:51, 11.03s/it][2022-12-21 01:38:03,752] [INFO] [timer.py:197:stop] 0/8946, RunningAvgSamplesPerSec=5.880255192035947, CurrSamplesPerSec=5.289065525072967, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4420/5000 [15:15:44<1:46:48, 11.05s/it][2022-12-21 01:38:14,829] [INFO] [timer.py:197:stop] 0/8948, RunningAvgSamplesPerSec=5.880246775259542, CurrSamplesPerSec=5.294686647431846, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4421/5000 [15:15:55<1:46:42, 11.06s/it][2022-12-21 01:38:25,917] [INFO] [timer.py:197:stop] 0/8950, RunningAvgSamplesPerSec=5.88023701480219, CurrSamplesPerSec=5.288457413367045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4422/5000 [15:16:06<1:46:33, 11.06s/it][2022-12-21 01:38:36,933] [INFO] [timer.py:197:stop] 0/8952, RunningAvgSamplesPerSec=5.880236014410982, CurrSamplesPerSec=5.336610263124945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4423/5000 [15:16:17<1:46:14, 11.05s/it][2022-12-21 01:38:47,969] [INFO] [timer.py:197:stop] 0/8954, RunningAvgSamplesPerSec=5.880232538243196, CurrSamplesPerSec=5.318243035458209, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4424/5000 [15:16:28<1:45:57, 11.04s/it][2022-12-21 01:38:58,947] [INFO] [timer.py:197:stop] 0/8956, RunningAvgSamplesPerSec=5.880235920909027, CurrSamplesPerSec=5.346678665227795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 88%|████████▊ | 4425/5000 [15:16:39<1:45:42, 11.03s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.1777777777777778e-06, 'epoch': 107.92}

 88%|████████▊ | 4425/5000 [15:16:39<1:45:42, 11.03s/it][2022-12-21 01:39:09,985] [INFO] [timer.py:197:stop] 0/8958, RunningAvgSamplesPerSec=5.880232615366944, CurrSamplesPerSec=5.328552297319036, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4426/5000 [15:16:50<1:45:30, 11.03s/it][2022-12-21 01:39:20,979] [INFO] [logging.py:68:log_dist] [Rank 0] step=4480, skipped=7, lr=[1.1733333333333335e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:39:20,980] [INFO] [timer.py:197:stop] 0/8960, RunningAvgSamplesPerSec=5.880234089778217, CurrSamplesPerSec=5.353798363254468, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4427/5000 [15:17:01<1:45:06, 11.01s/it][2022-12-21 01:39:31,992] [INFO] [timer.py:197:stop] 0/8962, RunningAvgSamplesPerSec=5.880233365884771, CurrSamplesPerSec=5.303484114547555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4428/5000 [15:17:12<1:44:58, 11.01s/it][2022-12-21 01:39:42,060] [INFO] [timer.py:197:stop] 0/8964, RunningAvgSamplesPerSec=5.880345141562016, CurrSamplesPerSec=6.283278509372505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 01:39:53,046] [INFO] [timer.py:197:stop] 0/8966, RunningAvgSamplesPerSec=5.880347378546373, CurrSamplesPerSec=5.3268327569644, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4429/5000 [15:17:28<1:59:24, 12.55s/it][2022-12-21 01:40:04,119] [INFO] [timer.py:197:stop] 0/8968, RunningAvgSamplesPerSec=5.880339378429554, CurrSamplesPerSec=5.284863908513213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4430/5000 [15:17:39<1:54:59, 12.10s/it][2022-12-21 01:40:15,166] [INFO] [timer.py:197:stop] 0/8970, RunningAvgSamplesPerSec=5.880334808896696, CurrSamplesPerSec=5.3276499870795755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4431/5000 [15:17:50<1:51:46, 11.79s/it][2022-12-21 01:40:26,167] [INFO] [timer.py:197:stop] 0/8972, RunningAvgSamplesPerSec=5.880335586985099, CurrSamplesPerSec=5.332013139173579, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4432/5000 [15:18:01<1:49:21, 11.55s/it][2022-12-21 01:40:37,214] [INFO] [timer.py:197:stop] 0/8974, RunningAvgSamplesPerSec=5.880331912248216, CurrSamplesPerSec=5.341967019422092, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4433/5000 [15:18:12<1:47:43, 11.40s/it][2022-12-21 01:40:48,288] [INFO] [timer.py:197:stop] 0/8976, RunningAvgSamplesPerSec=5.880323890598916, CurrSamplesPerSec=5.294927064715266, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4434/5000 [15:18:23<1:46:37, 11.30s/it][2022-12-21 01:40:59,325] [INFO] [timer.py:197:stop] 0/8978, RunningAvgSamplesPerSec=5.880320441813125, CurrSamplesPerSec=5.33420174219361, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4435/5000 [15:18:34<1:45:40, 11.22s/it][2022-12-21 01:41:10,367] [INFO] [logging.py:68:log_dist] [Rank 0] step=4490, skipped=7, lr=[1.151111111111111e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:41:10,368] [INFO] [timer.py:197:stop] 0/8980, RunningAvgSamplesPerSec=5.8803161155711505, CurrSamplesPerSec=5.298192090622961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4436/5000 [15:18:45<1:44:59, 11.17s/it][2022-12-21 01:41:21,384] [INFO] [timer.py:197:stop] 0/8982, RunningAvgSamplesPerSec=5.880315031593987, CurrSamplesPerSec=5.298366940820282, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▊ | 4437/5000 [15:18:56<1:44:22, 11.12s/it][2022-12-21 01:41:32,410] [INFO] [timer.py:197:stop] 0/8984, RunningAvgSamplesPerSec=5.8803127656037955, CurrSamplesPerSec=5.3319085008037925, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4438/5000 [15:19:08<1:43:54, 11.09s/it][2022-12-21 01:41:43,410] [INFO] [timer.py:197:stop] 0/8986, RunningAvgSamplesPerSec=5.880313567224994, CurrSamplesPerSec=5.329954382093274, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4439/5000 [15:19:19<1:43:27, 11.07s/it][2022-12-21 01:41:54,431] [INFO] [timer.py:197:stop] 0/8988, RunningAvgSamplesPerSec=5.880311845690864, CurrSamplesPerSec=5.341245078754454, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4440/5000 [15:19:30<1:43:09, 11.05s/it][2022-12-21 01:42:05,489] [INFO] [timer.py:197:stop] 0/8990, RunningAvgSamplesPerSec=5.8803058148707885, CurrSamplesPerSec=5.317468083261185, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4441/5000 [15:19:41<1:42:59, 11.05s/it][2022-12-21 01:42:16,543] [INFO] [timer.py:197:stop] 0/8992, RunningAvgSamplesPerSec=5.8803002922257015, CurrSamplesPerSec=5.318380856484618, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4442/5000 [15:19:52<1:42:48, 11.05s/it][2022-12-21 01:42:27,568] [INFO] [timer.py:197:stop] 0/8994, RunningAvgSamplesPerSec=5.88029814111771, CurrSamplesPerSec=5.337501177614581, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4443/5000 [15:20:03<1:42:32, 11.05s/it][2022-12-21 01:42:38,582] [INFO] [timer.py:197:stop] 0/8996, RunningAvgSamplesPerSec=5.880300130620294, CurrSamplesPerSec=5.341804161814273, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4444/5000 [15:20:14<1:42:15, 11.04s/it][2022-12-21 01:42:49,621] [INFO] [timer.py:197:stop] 0/8998, RunningAvgSamplesPerSec=5.8802962420489395, CurrSamplesPerSec=5.322498337831215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4445/5000 [15:20:25<1:42:05, 11.04s/it][2022-12-21 01:43:00,677] [INFO] [logging.py:68:log_dist] [Rank 0] step=4500, skipped=7, lr=[1.128888888888889e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:43:00,678] [INFO] [timer.py:197:stop] 0/9000, RunningAvgSamplesPerSec=5.880290152687617, CurrSamplesPerSec=5.29805782354453, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4446/5000 [15:20:36<1:41:57, 11.04s/it][2022-12-21 01:43:11,744] [INFO] [timer.py:197:stop] 0/9002, RunningAvgSamplesPerSec=5.88028292634447, CurrSamplesPerSec=5.310541274469182, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4447/5000 [15:20:47<1:41:50, 11.05s/it][2022-12-21 01:43:22,773] [INFO] [timer.py:197:stop] 0/9004, RunningAvgSamplesPerSec=5.880280277839604, CurrSamplesPerSec=5.324671530599606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4448/5000 [15:20:58<1:41:35, 11.04s/it][2022-12-21 01:43:33,778] [INFO] [timer.py:197:stop] 0/9006, RunningAvgSamplesPerSec=5.880280275706951, CurrSamplesPerSec=5.318105010855505, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4449/5000 [15:21:09<1:41:18, 11.03s/it][2022-12-21 01:43:44,777] [INFO] [timer.py:197:stop] 0/9008, RunningAvgSamplesPerSec=5.880281269400823, CurrSamplesPerSec=5.363377279379024, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4450/5000 [15:21:20<1:41:02, 11.02s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.12e-06, 'epoch': 108.53}

 89%|████████▉ | 4450/5000 [15:21:20<1:41:02, 11.02s/it][2022-12-21 01:43:55,800] [INFO] [timer.py:197:stop] 0/9010, RunningAvgSamplesPerSec=5.8802794726331555, CurrSamplesPerSec=5.334803880632716, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4451/5000 [15:21:31<1:40:51, 11.02s/it][2022-12-21 01:44:06,779] [INFO] [timer.py:197:stop] 0/9012, RunningAvgSamplesPerSec=5.880282845554919, CurrSamplesPerSec=5.36475979842679, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4452/5000 [15:21:42<1:40:33, 11.01s/it][2022-12-21 01:44:17,726] [INFO] [timer.py:197:stop] 0/9014, RunningAvgSamplesPerSec=5.8802900798659765, CurrSamplesPerSec=5.351650413208872, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4453/5000 [15:21:53<1:40:11, 10.99s/it][2022-12-21 01:44:28,680] [INFO] [timer.py:197:stop] 0/9016, RunningAvgSamplesPerSec=5.880296486377608, CurrSamplesPerSec=5.368660372563817, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4454/5000 [15:22:04<1:39:54, 10.98s/it][2022-12-21 01:44:39,680] [INFO] [timer.py:197:stop] 0/9018, RunningAvgSamplesPerSec=5.880297384226967, CurrSamplesPerSec=5.346908064809192, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4455/5000 [15:22:15<1:39:47, 10.99s/it][2022-12-21 01:44:50,665] [INFO] [logging.py:68:log_dist] [Rank 0] step=4510, skipped=7, lr=[1.1066666666666667e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:44:50,667] [INFO] [timer.py:197:stop] 0/9020, RunningAvgSamplesPerSec=5.88029981660445, CurrSamplesPerSec=5.334981367792814, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4456/5000 [15:22:26<1:39:36, 10.99s/it][2022-12-21 01:45:01,701] [INFO] [timer.py:197:stop] 0/9022, RunningAvgSamplesPerSec=5.880296565918671, CurrSamplesPerSec=5.314511939818211, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4457/5000 [15:22:37<1:39:33, 11.00s/it][2022-12-21 01:45:12,695] [INFO] [timer.py:197:stop] 0/9024, RunningAvgSamplesPerSec=5.880298121837218, CurrSamplesPerSec=5.303584915814932, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4458/5000 [15:22:48<1:39:21, 11.00s/it][2022-12-21 01:45:23,677] [INFO] [timer.py:197:stop] 0/9026, RunningAvgSamplesPerSec=5.880301194614621, CurrSamplesPerSec=5.321048066781988, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4459/5000 [15:22:59<1:39:07, 10.99s/it][2022-12-21 01:45:34,734] [INFO] [timer.py:197:stop] 0/9028, RunningAvgSamplesPerSec=5.88029518625755, CurrSamplesPerSec=5.304463788557064, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4460/5000 [15:23:10<1:39:06, 11.01s/it][2022-12-21 01:45:45,710] [INFO] [timer.py:197:stop] 0/9030, RunningAvgSamplesPerSec=5.880298884692065, CurrSamplesPerSec=5.373287375775656, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4461/5000 [15:23:21<1:38:49, 11.00s/it][2022-12-21 01:45:56,647] [INFO] [timer.py:197:stop] 0/9032, RunningAvgSamplesPerSec=5.8803073328253, CurrSamplesPerSec=5.338945141509502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4462/5000 [15:23:32<1:38:28, 10.98s/it][2022-12-21 01:46:07,639] [INFO] [timer.py:197:stop] 0/9034, RunningAvgSamplesPerSec=5.88030919820418, CurrSamplesPerSec=5.354824702126351, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4463/5000 [15:23:43<1:38:18, 10.99s/it][2022-12-21 01:46:18,671] [INFO] [timer.py:197:stop] 0/9036, RunningAvgSamplesPerSec=5.8803061779804775, CurrSamplesPerSec=5.332678557366441, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4464/5000 [15:23:54<1:38:15, 11.00s/it][2022-12-21 01:46:29,688] [INFO] [timer.py:197:stop] 0/9038, RunningAvgSamplesPerSec=5.880304959770268, CurrSamplesPerSec=5.322764084971344, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4465/5000 [15:24:05<1:38:07, 11.00s/it][2022-12-21 01:46:40,664] [INFO] [logging.py:68:log_dist] [Rank 0] step=4520, skipped=7, lr=[1.0844444444444446e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:46:40,665] [INFO] [timer.py:197:stop] 0/9040, RunningAvgSamplesPerSec=5.880308649653313, CurrSamplesPerSec=5.357850899367342, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4466/5000 [15:24:16<1:37:52, 11.00s/it][2022-12-21 01:46:51,613] [INFO] [timer.py:197:stop] 0/9042, RunningAvgSamplesPerSec=5.880315700622892, CurrSamplesPerSec=5.345496621650281, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4467/5000 [15:24:27<1:37:33, 10.98s/it][2022-12-21 01:47:02,553] [INFO] [timer.py:197:stop] 0/9044, RunningAvgSamplesPerSec=5.880323750764969, CurrSamplesPerSec=5.3944566457921, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4468/5000 [15:24:38<1:37:15, 10.97s/it][2022-12-21 01:47:13,494] [INFO] [timer.py:197:stop] 0/9046, RunningAvgSamplesPerSec=5.880331665819761, CurrSamplesPerSec=5.353449220067183, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4469/5000 [15:24:49<1:37:00, 10.96s/it][2022-12-21 01:47:23,566] [INFO] [timer.py:197:stop] 0/9048, RunningAvgSamplesPerSec=5.880442142739665, CurrSamplesPerSec=5.335131933662303, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4470/5000 [15:25:04<1:47:34, 12.18s/it][2022-12-21 01:47:34,597] [INFO] [timer.py:197:stop] 0/9050, RunningAvgSamplesPerSec=5.880439138327344, CurrSamplesPerSec=5.31464241239591, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4471/5000 [15:25:15<1:44:19, 11.83s/it][2022-12-21 01:47:45,580] [INFO] [timer.py:197:stop] 0/9052, RunningAvgSamplesPerSec=5.88044180438825, CurrSamplesPerSec=5.3541182909981035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4472/5000 [15:25:26<1:41:54, 11.58s/it][2022-12-21 01:47:56,555] [INFO] [timer.py:197:stop] 0/9054, RunningAvgSamplesPerSec=5.880445610031862, CurrSamplesPerSec=5.372273946099674, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4473/5000 [15:25:37<1:40:04, 11.39s/it][2022-12-21 01:48:07,598] [INFO] [timer.py:197:stop] 0/9056, RunningAvgSamplesPerSec=5.880441239572259, CurrSamplesPerSec=5.293550443830425, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 89%|████████▉ | 4474/5000 [15:25:48<1:38:59, 11.29s/it][2022-12-21 01:48:18,642] [INFO] [timer.py:197:stop] 0/9058, RunningAvgSamplesPerSec=5.880436822230419, CurrSamplesPerSec=5.302940775792831, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4475/5000 [15:25:59<1:38:02, 11.20s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.0644444444444445e-06, 'epoch': 109.14}

 90%|████████▉ | 4475/5000 [15:25:59<1:38:02, 11.20s/it][2022-12-21 01:48:29,590] [INFO] [logging.py:68:log_dist] [Rank 0] step=4530, skipped=7, lr=[1.0622222222222222e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:48:29,591] [INFO] [timer.py:197:stop] 0/9060, RunningAvgSamplesPerSec=5.880443546648543, CurrSamplesPerSec=5.346801563019808, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4476/5000 [15:26:10<1:37:13, 11.13s/it][2022-12-21 01:48:40,603] [INFO] [timer.py:197:stop] 0/9062, RunningAvgSamplesPerSec=5.880443118701863, CurrSamplesPerSec=5.30785834572428, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4477/5000 [15:26:21<1:36:42, 11.10s/it][2022-12-21 01:48:51,574] [INFO] [timer.py:197:stop] 0/9064, RunningAvgSamplesPerSec=5.8804472689290295, CurrSamplesPerSec=5.34007860894287, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4478/5000 [15:26:32<1:36:16, 11.07s/it][2022-12-21 01:49:02,578] [INFO] [timer.py:197:stop] 0/9066, RunningAvgSamplesPerSec=5.880447466225641, CurrSamplesPerSec=5.332307589012845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4479/5000 [15:26:43<1:35:49, 11.04s/it][2022-12-21 01:49:13,508] [INFO] [timer.py:197:stop] 0/9068, RunningAvgSamplesPerSec=5.88045647482423, CurrSamplesPerSec=5.363792024575232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4480/5000 [15:26:54<1:35:22, 11.01s/it][2022-12-21 01:49:24,498] [INFO] [timer.py:197:stop] 0/9070, RunningAvgSamplesPerSec=5.880458259339314, CurrSamplesPerSec=5.313922578218566, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4481/5000 [15:27:05<1:35:17, 11.02s/it][2022-12-21 01:49:35,542] [INFO] [timer.py:197:stop] 0/9072, RunningAvgSamplesPerSec=5.880453812257237, CurrSamplesPerSec=5.313910375753905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4482/5000 [15:27:16<1:35:08, 11.02s/it][2022-12-21 01:49:46,576] [INFO] [timer.py:197:stop] 0/9074, RunningAvgSamplesPerSec=5.88045063929097, CurrSamplesPerSec=5.311650099831585, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4483/5000 [15:27:27<1:35:01, 11.03s/it][2022-12-21 01:49:57,608] [INFO] [timer.py:197:stop] 0/9076, RunningAvgSamplesPerSec=5.880447688426041, CurrSamplesPerSec=5.324249506917437, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4484/5000 [15:27:38<1:34:46, 11.02s/it][2022-12-21 01:50:08,624] [INFO] [timer.py:197:stop] 0/9078, RunningAvgSamplesPerSec=5.880446539948908, CurrSamplesPerSec=5.3093709531036, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4485/5000 [15:27:49<1:34:37, 11.02s/it][2022-12-21 01:50:19,661] [INFO] [logging.py:68:log_dist] [Rank 0] step=4540, skipped=7, lr=[1.04e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:50:19,663] [INFO] [timer.py:197:stop] 0/9080, RunningAvgSamplesPerSec=5.880442793430431, CurrSamplesPerSec=5.307689795040298, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4486/5000 [15:28:00<1:34:25, 11.02s/it][2022-12-21 01:50:30,625] [INFO] [timer.py:197:stop] 0/9082, RunningAvgSamplesPerSec=5.880448267816349, CurrSamplesPerSec=5.355936069424194, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4487/5000 [15:28:11<1:34:12, 11.02s/it][2022-12-21 01:50:41,684] [INFO] [timer.py:197:stop] 0/9084, RunningAvgSamplesPerSec=5.880441980253298, CurrSamplesPerSec=5.312055201348783, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4488/5000 [15:28:22<1:34:03, 11.02s/it][2022-12-21 01:50:52,716] [INFO] [timer.py:197:stop] 0/9086, RunningAvgSamplesPerSec=5.880438894218136, CurrSamplesPerSec=5.3120036930567665, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4489/5000 [15:28:33<1:33:57, 11.03s/it][2022-12-21 01:51:03,763] [INFO] [timer.py:197:stop] 0/9088, RunningAvgSamplesPerSec=5.880434080770759, CurrSamplesPerSec=5.316222270175449, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4490/5000 [15:28:44<1:33:40, 11.02s/it][2022-12-21 01:51:14,757] [INFO] [timer.py:197:stop] 0/9090, RunningAvgSamplesPerSec=5.880435514913358, CurrSamplesPerSec=5.315166892610625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4491/5000 [15:28:55<1:33:31, 11.02s/it][2022-12-21 01:51:25,788] [INFO] [timer.py:197:stop] 0/9092, RunningAvgSamplesPerSec=5.880432551923307, CurrSamplesPerSec=5.319315446175143, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4492/5000 [15:29:06<1:33:23, 11.03s/it][2022-12-21 01:51:36,845] [INFO] [timer.py:197:stop] 0/9094, RunningAvgSamplesPerSec=5.880426543341033, CurrSamplesPerSec=5.3059515979280905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4493/5000 [15:29:17<1:33:01, 11.01s/it][2022-12-21 01:51:47,809] [INFO] [timer.py:197:stop] 0/9096, RunningAvgSamplesPerSec=5.880431608737186, CurrSamplesPerSec=5.304834666744661, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4494/5000 [15:29:28<1:32:55, 11.02s/it][2022-12-21 01:51:58,806] [INFO] [timer.py:197:stop] 0/9098, RunningAvgSamplesPerSec=5.880432549942161, CurrSamplesPerSec=5.341934489628464, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4495/5000 [15:29:39<1:32:48, 11.03s/it][2022-12-21 01:52:09,887] [INFO] [logging.py:68:log_dist] [Rank 0] step=4550, skipped=7, lr=[1.0177777777777778e-06], mom=[[0.9, 0.999]]
[2022-12-21 01:52:09,888] [INFO] [timer.py:197:stop] 0/9100, RunningAvgSamplesPerSec=5.880423577772919, CurrSamplesPerSec=5.308138377624003, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4496/5000 [15:29:50<1:32:41, 11.03s/it][2022-12-21 01:52:20,895] [INFO] [timer.py:197:stop] 0/9102, RunningAvgSamplesPerSec=5.880423563447671, CurrSamplesPerSec=5.350106372135575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4497/5000 [15:30:01<1:32:21, 11.02s/it][2022-12-21 01:52:31,907] [INFO] [timer.py:197:stop] 0/9104, RunningAvgSamplesPerSec=5.880422852134698, CurrSamplesPerSec=5.3170299285873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4498/5000 [15:30:12<1:32:09, 11.01s/it][2022-12-21 01:52:42,883] [INFO] [timer.py:197:stop] 0/9106, RunningAvgSamplesPerSec=5.880426467413257, CurrSamplesPerSec=5.347336457858315, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|████████▉ | 4499/5000 [15:30:23<1:31:49, 11.00s/it][2022-12-21 01:52:53,839] [INFO] [timer.py:197:stop] 0/9108, RunningAvgSamplesPerSec=5.880432417836862, CurrSamplesPerSec=5.345937351420396, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4500/5000 [15:30:34<1:31:39, 11.00s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 1.008888888888889e-06, 'epoch': 109.75}

 90%|█████████ | 4500/5000 [15:30:34<1:31:39, 11.00s/it][2022-12-21 01:53:04,865] [INFO] [timer.py:197:stop] 0/9110, RunningAvgSamplesPerSec=5.880430375024078, CurrSamplesPerSec=5.3315990579172725, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4501/5000 [15:30:45<1:31:24, 10.99s/it][2022-12-21 01:53:15,850] [INFO] [timer.py:197:stop] 0/9112, RunningAvgSamplesPerSec=5.880432702362991, CurrSamplesPerSec=5.315968334955242, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4502/5000 [15:30:56<1:31:19, 11.00s/it][2022-12-21 01:53:26,902] [INFO] [timer.py:197:stop] 0/9114, RunningAvgSamplesPerSec=5.880427067632778, CurrSamplesPerSec=5.297638753258436, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4503/5000 [15:31:07<1:31:16, 11.02s/it][2022-12-21 01:53:37,892] [INFO] [timer.py:197:stop] 0/9116, RunningAvgSamplesPerSec=5.880428941391807, CurrSamplesPerSec=5.3543357270010095, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4504/5000 [15:31:18<1:30:58, 11.00s/it][2022-12-21 01:53:48,902] [INFO] [timer.py:197:stop] 0/9118, RunningAvgSamplesPerSec=5.880428502210159, CurrSamplesPerSec=5.322274826827844, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4505/5000 [15:31:29<1:30:50, 11.01s/it][2022-12-21 01:53:59,938] [INFO] [logging.py:68:log_dist] [Rank 0] step=4560, skipped=7, lr=[9.955555555555556e-07], mom=[[0.9, 0.999]]
[2022-12-21 01:53:59,940] [INFO] [timer.py:197:stop] 0/9120, RunningAvgSamplesPerSec=5.880424646723352, CurrSamplesPerSec=5.308895283333363, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4506/5000 [15:31:40<1:30:40, 11.01s/it][2022-12-21 01:54:10,959] [INFO] [timer.py:197:stop] 0/9122, RunningAvgSamplesPerSec=5.880423075584787, CurrSamplesPerSec=5.31188491243612, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4507/5000 [15:31:51<1:30:36, 11.03s/it][2022-12-21 01:54:21,992] [INFO] [timer.py:197:stop] 0/9124, RunningAvgSamplesPerSec=5.880419900996102, CurrSamplesPerSec=5.334929201882353, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4508/5000 [15:32:02<1:30:17, 11.01s/it][2022-12-21 01:54:32,995] [INFO] [timer.py:197:stop] 0/9126, RunningAvgSamplesPerSec=5.880420261742048, CurrSamplesPerSec=5.310945366256265, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4509/5000 [15:32:13<1:30:07, 11.01s/it][2022-12-21 01:54:43,982] [INFO] [timer.py:197:stop] 0/9128, RunningAvgSamplesPerSec=5.880422626054864, CurrSamplesPerSec=5.339304714647681, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4510/5000 [15:32:24<1:29:56, 11.01s/it][2022-12-21 01:54:54,092] [INFO] [timer.py:197:stop] 0/9130, RunningAvgSamplesPerSec=5.880527447077867, CurrSamplesPerSec=6.2675497047472755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 01:55:05,154] [INFO] [timer.py:197:stop] 0/9132, RunningAvgSamplesPerSec=5.880520633297506, CurrSamplesPerSec=5.3005903848695795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4511/5000 [15:32:40<1:42:29, 12.57s/it][2022-12-21 01:55:16,213] [INFO] [timer.py:197:stop] 0/9134, RunningAvgSamplesPerSec=5.880514647617238, CurrSamplesPerSec=5.313718930628229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4512/5000 [15:32:51<1:38:34, 12.12s/it][2022-12-21 01:55:27,262] [INFO] [timer.py:197:stop] 0/9136, RunningAvgSamplesPerSec=5.880509566493539, CurrSamplesPerSec=5.284806475434704, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4513/5000 [15:33:02<1:35:46, 11.80s/it][2022-12-21 01:55:38,301] [INFO] [timer.py:197:stop] 0/9138, RunningAvgSamplesPerSec=5.880505794490363, CurrSamplesPerSec=5.3278105023473605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4514/5000 [15:33:13<1:33:43, 11.57s/it][2022-12-21 01:55:49,317] [INFO] [logging.py:68:log_dist] [Rank 0] step=4570, skipped=7, lr=[9.733333333333333e-07], mom=[[0.9, 0.999]]
[2022-12-21 01:55:49,319] [INFO] [timer.py:197:stop] 0/9140, RunningAvgSamplesPerSec=5.880504475731382, CurrSamplesPerSec=5.311544997961143, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4515/5000 [15:33:24<1:32:11, 11.40s/it][2022-12-21 01:56:00,315] [INFO] [timer.py:197:stop] 0/9142, RunningAvgSamplesPerSec=5.880505728686221, CurrSamplesPerSec=5.322365579740216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4516/5000 [15:33:35<1:31:00, 11.28s/it][2022-12-21 01:56:11,371] [INFO] [timer.py:197:stop] 0/9144, RunningAvgSamplesPerSec=5.88049995316461, CurrSamplesPerSec=5.309683072382902, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4517/5000 [15:33:46<1:30:16, 11.21s/it][2022-12-21 01:56:22,485] [INFO] [timer.py:197:stop] 0/9146, RunningAvgSamplesPerSec=5.880487383736593, CurrSamplesPerSec=5.232245869040384, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4518/5000 [15:33:58<1:29:50, 11.18s/it][2022-12-21 01:56:33,486] [INFO] [timer.py:197:stop] 0/9148, RunningAvgSamplesPerSec=5.880488057383731, CurrSamplesPerSec=5.308947991325011, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4519/5000 [15:34:09<1:29:13, 11.13s/it][2022-12-21 01:56:44,459] [INFO] [timer.py:197:stop] 0/9150, RunningAvgSamplesPerSec=5.880492117065759, CurrSamplesPerSec=5.336426090181879, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4520/5000 [15:34:20<1:28:39, 11.08s/it][2022-12-21 01:56:55,478] [INFO] [timer.py:197:stop] 0/9152, RunningAvgSamplesPerSec=5.880490665904338, CurrSamplesPerSec=5.297541523371702, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4521/5000 [15:34:31<1:28:19, 11.06s/it][2022-12-21 01:57:06,523] [INFO] [timer.py:197:stop] 0/9154, RunningAvgSamplesPerSec=5.880486153541542, CurrSamplesPerSec=5.304776798785327, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4522/5000 [15:34:42<1:28:05, 11.06s/it][2022-12-21 01:57:17,556] [INFO] [timer.py:197:stop] 0/9156, RunningAvgSamplesPerSec=5.880483049125236, CurrSamplesPerSec=5.32392743675845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4523/5000 [15:34:53<1:27:51, 11.05s/it][2022-12-21 01:57:28,641] [INFO] [timer.py:197:stop] 0/9158, RunningAvgSamplesPerSec=5.880473824586998, CurrSamplesPerSec=5.294529792668037, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4524/5000 [15:35:04<1:27:44, 11.06s/it][2022-12-21 01:57:39,697] [INFO] [logging.py:68:log_dist] [Rank 0] step=4580, skipped=7, lr=[9.511111111111111e-07], mom=[[0.9, 0.999]]
[2022-12-21 01:57:39,698] [INFO] [timer.py:197:stop] 0/9160, RunningAvgSamplesPerSec=5.880467916294156, CurrSamplesPerSec=5.300919477464728, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 90%|█████████ | 4525/5000 [15:35:15<1:27:33, 11.06s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 9.511111111111111e-07, 'epoch': 110.36}

 90%|█████████ | 4525/5000 [15:35:15<1:27:33, 11.06s/it][2022-12-21 01:57:50,684] [INFO] [timer.py:197:stop] 0/9162, RunningAvgSamplesPerSec=5.880470719592, CurrSamplesPerSec=5.345484273746796, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4526/5000 [15:35:26<1:27:11, 11.04s/it][2022-12-21 01:58:01,778] [INFO] [timer.py:197:stop] 0/9164, RunningAvgSamplesPerSec=5.8804605196927024, CurrSamplesPerSec=5.283858385788975, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4527/5000 [15:35:37<1:27:08, 11.05s/it][2022-12-21 01:58:12,826] [INFO] [timer.py:197:stop] 0/9166, RunningAvgSamplesPerSec=5.880455690449079, CurrSamplesPerSec=5.320475814456463, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4528/5000 [15:35:48<1:26:56, 11.05s/it][2022-12-21 01:58:23,851] [INFO] [timer.py:197:stop] 0/9168, RunningAvgSamplesPerSec=5.880453431812182, CurrSamplesPerSec=5.3264192685188965, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4529/5000 [15:35:59<1:26:41, 11.04s/it][2022-12-21 01:58:34,885] [INFO] [timer.py:197:stop] 0/9170, RunningAvgSamplesPerSec=5.880450222489306, CurrSamplesPerSec=5.315511270583549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4530/5000 [15:36:10<1:26:29, 11.04s/it][2022-12-21 01:58:45,872] [INFO] [timer.py:197:stop] 0/9172, RunningAvgSamplesPerSec=5.880452560630593, CurrSamplesPerSec=5.3689595282723985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4531/5000 [15:36:21<1:26:10, 11.02s/it][2022-12-21 01:58:56,936] [INFO] [timer.py:197:stop] 0/9174, RunningAvgSamplesPerSec=5.880445866406771, CurrSamplesPerSec=5.29041041106696, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4532/5000 [15:36:32<1:26:05, 11.04s/it][2022-12-21 01:59:07,996] [INFO] [timer.py:197:stop] 0/9176, RunningAvgSamplesPerSec=5.880439635938803, CurrSamplesPerSec=5.278972499950049, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4533/5000 [15:36:43<1:25:57, 11.04s/it][2022-12-21 01:59:18,972] [INFO] [timer.py:197:stop] 0/9178, RunningAvgSamplesPerSec=5.880443260907056, CurrSamplesPerSec=5.3331896502740035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4534/5000 [15:36:54<1:25:36, 11.02s/it][2022-12-21 01:59:29,962] [INFO] [logging.py:68:log_dist] [Rank 0] step=4590, skipped=7, lr=[9.288888888888889e-07], mom=[[0.9, 0.999]]
[2022-12-21 01:59:29,964] [INFO] [timer.py:197:stop] 0/9180, RunningAvgSamplesPerSec=5.880445155837406, CurrSamplesPerSec=5.3199323625461705, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4535/5000 [15:37:05<1:25:21, 11.01s/it][2022-12-21 01:59:40,976] [INFO] [timer.py:197:stop] 0/9182, RunningAvgSamplesPerSec=5.880444547931888, CurrSamplesPerSec=5.313725452158301, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4536/5000 [15:37:16<1:25:10, 11.01s/it][2022-12-21 01:59:52,011] [INFO] [timer.py:197:stop] 0/9184, RunningAvgSamplesPerSec=5.880441373646404, CurrSamplesPerSec=5.32735435976437, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4537/5000 [15:37:27<1:25:02, 11.02s/it][2022-12-21 02:00:03,011] [INFO] [timer.py:197:stop] 0/9186, RunningAvgSamplesPerSec=5.880442397883138, CurrSamplesPerSec=5.341489955484841, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4538/5000 [15:37:38<1:24:48, 11.01s/it][2022-12-21 02:00:13,940] [INFO] [timer.py:197:stop] 0/9188, RunningAvgSamplesPerSec=5.880451457383316, CurrSamplesPerSec=5.369275256383033, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4539/5000 [15:37:49<1:24:25, 10.99s/it][2022-12-21 02:00:24,926] [INFO] [timer.py:197:stop] 0/9190, RunningAvgSamplesPerSec=5.88045399071944, CurrSamplesPerSec=5.351170764841647, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4540/5000 [15:38:00<1:24:14, 10.99s/it][2022-12-21 02:00:35,933] [INFO] [timer.py:197:stop] 0/9192, RunningAvgSamplesPerSec=5.880453991418233, CurrSamplesPerSec=5.308394924531674, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4541/5000 [15:38:11<1:24:06, 10.99s/it][2022-12-21 02:00:46,974] [INFO] [timer.py:197:stop] 0/9194, RunningAvgSamplesPerSec=5.880450043362405, CurrSamplesPerSec=5.311240015880964, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4542/5000 [15:38:22<1:24:01, 11.01s/it][2022-12-21 02:00:57,965] [INFO] [timer.py:197:stop] 0/9196, RunningAvgSamplesPerSec=5.880451937251019, CurrSamplesPerSec=5.347196705988205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4543/5000 [15:38:33<1:23:48, 11.00s/it][2022-12-21 02:01:08,950] [INFO] [timer.py:197:stop] 0/9198, RunningAvgSamplesPerSec=5.880454453262519, CurrSamplesPerSec=5.363368706511553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4544/5000 [15:38:44<1:23:34, 11.00s/it][2022-12-21 02:01:19,842] [INFO] [logging.py:68:log_dist] [Rank 0] step=4600, skipped=7, lr=[9.066666666666668e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:01:19,843] [INFO] [timer.py:197:stop] 0/9200, RunningAvgSamplesPerSec=5.88046790636436, CurrSamplesPerSec=5.395881267214939, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4545/5000 [15:38:55<1:23:09, 10.97s/it][2022-12-21 02:01:30,809] [INFO] [timer.py:197:stop] 0/9202, RunningAvgSamplesPerSec=5.880472799880131, CurrSamplesPerSec=5.366177572068601, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4546/5000 [15:39:06<1:22:58, 10.97s/it][2022-12-21 02:01:41,777] [INFO] [timer.py:197:stop] 0/9204, RunningAvgSamplesPerSec=5.880477303583169, CurrSamplesPerSec=5.329611727917405, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4547/5000 [15:39:17<1:22:47, 10.97s/it][2022-12-21 02:01:52,767] [INFO] [timer.py:197:stop] 0/9206, RunningAvgSamplesPerSec=5.880479329077776, CurrSamplesPerSec=5.307589047563874, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4548/5000 [15:39:28<1:22:40, 10.97s/it][2022-12-21 02:02:03,790] [INFO] [timer.py:197:stop] 0/9208, RunningAvgSamplesPerSec=5.880477537573404, CurrSamplesPerSec=5.329611939549376, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4549/5000 [15:39:39<1:22:35, 10.99s/it][2022-12-21 02:02:14,751] [INFO] [timer.py:197:stop] 0/9210, RunningAvgSamplesPerSec=5.880483038815123, CurrSamplesPerSec=5.353364023229255, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4550/5000 [15:39:50<1:22:21, 10.98s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 8.955555555555557e-07, 'epoch': 110.96}

 91%|█████████ | 4550/5000 [15:39:50<1:22:21, 10.98s/it][2022-12-21 02:02:25,729] [INFO] [timer.py:197:stop] 0/9212, RunningAvgSamplesPerSec=5.880486687802154, CurrSamplesPerSec=5.365596001200104, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4551/5000 [15:40:01<1:22:09, 10.98s/it][2022-12-21 02:02:35,855] [INFO] [timer.py:197:stop] 0/9214, RunningAvgSamplesPerSec=5.880588804216937, CurrSamplesPerSec=5.316216374223607, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4552/5000 [15:40:16<1:31:07, 12.20s/it][2022-12-21 02:02:46,861] [INFO] [timer.py:197:stop] 0/9216, RunningAvgSamplesPerSec=5.880588794805165, CurrSamplesPerSec=5.3245530277054804, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4553/5000 [15:40:27<1:28:10, 11.84s/it][2022-12-21 02:02:57,849] [INFO] [timer.py:197:stop] 0/9218, RunningAvgSamplesPerSec=5.880590965366828, CurrSamplesPerSec=5.317940022906913, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4554/5000 [15:40:38<1:26:05, 11.58s/it][2022-12-21 02:03:08,797] [INFO] [logging.py:68:log_dist] [Rank 0] step=4610, skipped=7, lr=[8.844444444444446e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:03:08,799] [INFO] [timer.py:197:stop] 0/9220, RunningAvgSamplesPerSec=5.880597483116103, CurrSamplesPerSec=5.348682156375526, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4555/5000 [15:40:49<1:24:30, 11.40s/it][2022-12-21 02:03:19,799] [INFO] [timer.py:197:stop] 0/9222, RunningAvgSamplesPerSec=5.8805981401574, CurrSamplesPerSec=5.314558025367921, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4556/5000 [15:41:00<1:23:20, 11.26s/it][2022-12-21 02:03:30,757] [INFO] [timer.py:197:stop] 0/9224, RunningAvgSamplesPerSec=5.880603738513444, CurrSamplesPerSec=5.309416949534447, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4557/5000 [15:41:11<1:22:33, 11.18s/it][2022-12-21 02:03:41,719] [INFO] [timer.py:197:stop] 0/9226, RunningAvgSamplesPerSec=5.8806088706272215, CurrSamplesPerSec=5.335077644143741, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4558/5000 [15:41:22<1:21:54, 11.12s/it][2022-12-21 02:03:52,701] [INFO] [timer.py:197:stop] 0/9228, RunningAvgSamplesPerSec=5.880611627218014, CurrSamplesPerSec=5.325546207057345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4559/5000 [15:41:33<1:21:25, 11.08s/it][2022-12-21 02:04:03,665] [INFO] [timer.py:197:stop] 0/9230, RunningAvgSamplesPerSec=5.880616570351992, CurrSamplesPerSec=5.347002215924146, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4560/5000 [15:41:44<1:21:02, 11.05s/it][2022-12-21 02:04:14,694] [INFO] [timer.py:197:stop] 0/9232, RunningAvgSamplesPerSec=5.8806139242247335, CurrSamplesPerSec=5.31011182362186, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4561/5000 [15:41:55<1:20:52, 11.05s/it][2022-12-21 02:04:25,707] [INFO] [timer.py:197:stop] 0/9234, RunningAvgSamplesPerSec=5.88061308717108, CurrSamplesPerSec=5.347475577874399, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████ | 4562/5000 [15:42:06<1:20:35, 11.04s/it][2022-12-21 02:04:36,745] [INFO] [timer.py:197:stop] 0/9236, RunningAvgSamplesPerSec=5.880609431114901, CurrSamplesPerSec=5.323133515467272, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4563/5000 [15:42:17<1:20:21, 11.03s/it][2022-12-21 02:04:47,737] [INFO] [timer.py:197:stop] 0/9238, RunningAvgSamplesPerSec=5.88061097519919, CurrSamplesPerSec=5.344057197908337, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4564/5000 [15:42:28<1:19:59, 11.01s/it][2022-12-21 02:04:58,706] [INFO] [logging.py:68:log_dist] [Rank 0] step=4620, skipped=7, lr=[8.622222222222224e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:04:58,707] [INFO] [timer.py:197:stop] 0/9240, RunningAvgSamplesPerSec=5.880615181189549, CurrSamplesPerSec=5.328191632676143, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4565/5000 [15:42:39<1:19:44, 11.00s/it][2022-12-21 02:05:09,663] [INFO] [timer.py:197:stop] 0/9242, RunningAvgSamplesPerSec=5.88062103320555, CurrSamplesPerSec=5.347381197043217, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4566/5000 [15:42:50<1:19:31, 11.00s/it][2022-12-21 02:05:20,682] [INFO] [timer.py:197:stop] 0/9244, RunningAvgSamplesPerSec=5.880619488711999, CurrSamplesPerSec=5.317811284632525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4567/5000 [15:43:01<1:19:20, 10.99s/it][2022-12-21 02:05:31,648] [INFO] [timer.py:197:stop] 0/9246, RunningAvgSamplesPerSec=5.880624277501237, CurrSamplesPerSec=5.3393180960358375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4568/5000 [15:43:12<1:19:13, 11.00s/it][2022-12-21 02:05:42,664] [INFO] [timer.py:197:stop] 0/9248, RunningAvgSamplesPerSec=5.88062295969447, CurrSamplesPerSec=5.3458892295635465, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4569/5000 [15:43:23<1:19:02, 11.00s/it][2022-12-21 02:05:53,695] [INFO] [timer.py:197:stop] 0/9250, RunningAvgSamplesPerSec=5.880620046442602, CurrSamplesPerSec=5.321629937137331, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4570/5000 [15:43:34<1:18:53, 11.01s/it][2022-12-21 02:06:04,708] [INFO] [timer.py:197:stop] 0/9252, RunningAvgSamplesPerSec=5.8806191568524655, CurrSamplesPerSec=5.325209612757232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4571/5000 [15:43:45<1:18:40, 11.00s/it][2022-12-21 02:06:15,691] [INFO] [timer.py:197:stop] 0/9254, RunningAvgSamplesPerSec=5.880621792033226, CurrSamplesPerSec=5.335485478762584, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4572/5000 [15:43:56<1:18:27, 11.00s/it][2022-12-21 02:06:26,711] [INFO] [timer.py:197:stop] 0/9256, RunningAvgSamplesPerSec=5.880620106435126, CurrSamplesPerSec=5.31010405044518, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4573/5000 [15:44:07<1:18:15, 11.00s/it][2022-12-21 02:06:37,679] [INFO] [timer.py:197:stop] 0/9258, RunningAvgSamplesPerSec=5.880624626107431, CurrSamplesPerSec=5.330560007803354, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 91%|█████████▏| 4574/5000 [15:44:18<1:18:03, 10.99s/it][2022-12-21 02:06:48,673] [INFO] [logging.py:68:log_dist] [Rank 0] step=4630, skipped=7, lr=[8.400000000000001e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:06:48,675] [INFO] [timer.py:197:stop] 0/9260, RunningAvgSamplesPerSec=5.88062567702227, CurrSamplesPerSec=5.321498699279524, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4575/5000 [15:44:29<1:17:58, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 8.400000000000001e-07, 'epoch': 111.58}

 92%|█████████▏| 4575/5000 [15:44:29<1:17:58, 11.01s/it][2022-12-21 02:06:59,724] [INFO] [timer.py:197:stop] 0/9262, RunningAvgSamplesPerSec=5.880620863434008, CurrSamplesPerSec=5.315309606184809, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4576/5000 [15:44:40<1:17:51, 11.02s/it][2022-12-21 02:07:10,732] [INFO] [timer.py:197:stop] 0/9264, RunningAvgSamplesPerSec=5.88062063005357, CurrSamplesPerSec=5.344907817894291, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4577/5000 [15:44:51<1:17:40, 11.02s/it][2022-12-21 02:07:21,801] [INFO] [timer.py:197:stop] 0/9266, RunningAvgSamplesPerSec=5.880613316033379, CurrSamplesPerSec=5.299346817402509, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4578/5000 [15:45:02<1:17:34, 11.03s/it][2022-12-21 02:07:32,799] [INFO] [timer.py:197:stop] 0/9268, RunningAvgSamplesPerSec=5.880614300769703, CurrSamplesPerSec=5.350375096663765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4579/5000 [15:45:13<1:17:21, 11.02s/it][2022-12-21 02:07:43,787] [INFO] [timer.py:197:stop] 0/9270, RunningAvgSamplesPerSec=5.880616424227047, CurrSamplesPerSec=5.371789733976641, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4580/5000 [15:45:24<1:17:00, 11.00s/it][2022-12-21 02:07:54,717] [INFO] [timer.py:197:stop] 0/9272, RunningAvgSamplesPerSec=5.880625356419194, CurrSamplesPerSec=5.390039662962207, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4581/5000 [15:45:35<1:16:45, 10.99s/it][2022-12-21 02:08:05,744] [INFO] [timer.py:197:stop] 0/9274, RunningAvgSamplesPerSec=5.880622917200362, CurrSamplesPerSec=5.334644427926808, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4582/5000 [15:45:46<1:16:37, 11.00s/it][2022-12-21 02:08:16,805] [INFO] [timer.py:197:stop] 0/9276, RunningAvgSamplesPerSec=5.880616405448078, CurrSamplesPerSec=5.291830465365266, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4583/5000 [15:45:57<1:16:37, 11.02s/it][2022-12-21 02:08:27,871] [INFO] [timer.py:197:stop] 0/9278, RunningAvgSamplesPerSec=5.880609658836568, CurrSamplesPerSec=5.312225290926635, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4584/5000 [15:46:08<1:16:23, 11.02s/it][2022-12-21 02:08:38,873] [INFO] [logging.py:68:log_dist] [Rank 0] step=4640, skipped=7, lr=[8.177777777777779e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:08:38,874] [INFO] [timer.py:197:stop] 0/9280, RunningAvgSamplesPerSec=5.8806099725725245, CurrSamplesPerSec=5.31215065200807, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4585/5000 [15:46:19<1:16:10, 11.01s/it][2022-12-21 02:08:49,862] [INFO] [timer.py:197:stop] 0/9282, RunningAvgSamplesPerSec=5.880612144744779, CurrSamplesPerSec=5.324734480806193, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4586/5000 [15:46:30<1:15:56, 11.01s/it][2022-12-21 02:09:00,835] [INFO] [timer.py:197:stop] 0/9284, RunningAvgSamplesPerSec=5.88061606481585, CurrSamplesPerSec=5.340500383355103, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4587/5000 [15:46:41<1:15:46, 11.01s/it][2022-12-21 02:09:11,868] [INFO] [timer.py:197:stop] 0/9286, RunningAvgSamplesPerSec=5.880612881389406, CurrSamplesPerSec=5.324371164539262, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4588/5000 [15:46:52<1:15:40, 11.02s/it][2022-12-21 02:09:22,877] [INFO] [timer.py:197:stop] 0/9288, RunningAvgSamplesPerSec=5.880612506910815, CurrSamplesPerSec=5.356191913745254, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4589/5000 [15:47:03<1:15:24, 11.01s/it][2022-12-21 02:09:33,908] [INFO] [timer.py:197:stop] 0/9290, RunningAvgSamplesPerSec=5.880609641505174, CurrSamplesPerSec=5.316292390891075, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4590/5000 [15:47:14<1:15:17, 11.02s/it][2022-12-21 02:09:44,902] [INFO] [timer.py:197:stop] 0/9292, RunningAvgSamplesPerSec=5.8806111918783825, CurrSamplesPerSec=5.358036981939373, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4591/5000 [15:47:25<1:14:58, 11.00s/it][2022-12-21 02:09:55,880] [INFO] [timer.py:197:stop] 0/9294, RunningAvgSamplesPerSec=5.880614464272843, CurrSamplesPerSec=5.334252621874057, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4592/5000 [15:47:36<1:14:54, 11.02s/it][2022-12-21 02:10:06,063] [INFO] [timer.py:197:stop] 0/9296, RunningAvgSamplesPerSec=5.880708945399914, CurrSamplesPerSec=6.227135142718963, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 02:10:17,048] [INFO] [timer.py:197:stop] 0/9298, RunningAvgSamplesPerSec=5.880711170359993, CurrSamplesPerSec=5.3419938089613845, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4593/5000 [15:47:52<1:25:13, 12.56s/it][2022-12-21 02:10:28,065] [INFO] [logging.py:68:log_dist] [Rank 0] step=4650, skipped=7, lr=[7.955555555555557e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:10:28,067] [INFO] [timer.py:197:stop] 0/9300, RunningAvgSamplesPerSec=5.880709766705022, CurrSamplesPerSec=5.338996536475208, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4594/5000 [15:48:03<1:21:52, 12.10s/it][2022-12-21 02:10:39,110] [INFO] [timer.py:197:stop] 0/9302, RunningAvgSamplesPerSec=5.8807055410718725, CurrSamplesPerSec=5.305883008240703, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4595/5000 [15:48:14<1:19:32, 11.78s/it][2022-12-21 02:10:50,117] [INFO] [timer.py:197:stop] 0/9304, RunningAvgSamplesPerSec=5.88070564869224, CurrSamplesPerSec=5.312057724229564, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4596/5000 [15:48:25<1:17:46, 11.55s/it][2022-12-21 02:11:01,180] [INFO] [timer.py:197:stop] 0/9306, RunningAvgSamplesPerSec=5.880699168937078, CurrSamplesPerSec=5.311506111323249, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4597/5000 [15:48:36<1:16:35, 11.40s/it][2022-12-21 02:11:12,237] [INFO] [timer.py:197:stop] 0/9308, RunningAvgSamplesPerSec=5.8806931509574145, CurrSamplesPerSec=5.2975043051338355, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4598/5000 [15:48:47<1:15:42, 11.30s/it][2022-12-21 02:11:23,220] [INFO] [timer.py:197:stop] 0/9310, RunningAvgSamplesPerSec=5.880696019668126, CurrSamplesPerSec=5.324819191319168, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4599/5000 [15:48:58<1:14:53, 11.20s/it][2022-12-21 02:11:34,231] [INFO] [timer.py:197:stop] 0/9312, RunningAvgSamplesPerSec=5.880695272624321, CurrSamplesPerSec=5.2974313338145125, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4600/5000 [15:49:09<1:14:18, 11.15s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 7.822222222222223e-07, 'epoch': 112.19}

 92%|█████████▏| 4600/5000 [15:49:09<1:14:18, 11.15s/it][2022-12-21 02:11:45,229] [INFO] [timer.py:197:stop] 0/9314, RunningAvgSamplesPerSec=5.880696334476508, CurrSamplesPerSec=5.2993564422404, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4601/5000 [15:49:20<1:13:49, 11.10s/it][2022-12-21 02:11:56,256] [INFO] [timer.py:197:stop] 0/9316, RunningAvgSamplesPerSec=5.8806938485761355, CurrSamplesPerSec=5.350169711914274, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4602/5000 [15:49:31<1:13:29, 11.08s/it][2022-12-21 02:12:07,339] [INFO] [timer.py:197:stop] 0/9318, RunningAvgSamplesPerSec=5.880685202537343, CurrSamplesPerSec=5.311841816356127, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4603/5000 [15:49:42<1:13:18, 11.08s/it][2022-12-21 02:12:18,341] [INFO] [logging.py:68:log_dist] [Rank 0] step=4660, skipped=7, lr=[7.733333333333335e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:12:18,343] [INFO] [timer.py:197:stop] 0/9320, RunningAvgSamplesPerSec=5.880685432687263, CurrSamplesPerSec=5.312487279619281, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4604/5000 [15:49:53<1:12:58, 11.06s/it][2022-12-21 02:12:29,369] [INFO] [timer.py:197:stop] 0/9322, RunningAvgSamplesPerSec=5.8806832894527075, CurrSamplesPerSec=5.32899044890735, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4605/5000 [15:50:04<1:12:43, 11.05s/it][2022-12-21 02:12:40,386] [INFO] [timer.py:197:stop] 0/9324, RunningAvgSamplesPerSec=5.88068180333802, CurrSamplesPerSec=5.3088287173653494, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4606/5000 [15:50:15<1:12:29, 11.04s/it][2022-12-21 02:12:51,419] [INFO] [timer.py:197:stop] 0/9326, RunningAvgSamplesPerSec=5.880678782547304, CurrSamplesPerSec=5.329206906724594, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4607/5000 [15:50:27<1:12:17, 11.04s/it][2022-12-21 02:13:02,407] [INFO] [timer.py:197:stop] 0/9328, RunningAvgSamplesPerSec=5.880680749026344, CurrSamplesPerSec=5.319663735438205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4608/5000 [15:50:38<1:12:00, 11.02s/it][2022-12-21 02:13:13,466] [INFO] [timer.py:197:stop] 0/9330, RunningAvgSamplesPerSec=5.880674515019637, CurrSamplesPerSec=5.30754035432129, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4609/5000 [15:50:49<1:11:54, 11.03s/it][2022-12-21 02:13:24,495] [INFO] [timer.py:197:stop] 0/9332, RunningAvgSamplesPerSec=5.880671873900263, CurrSamplesPerSec=5.339664973516253, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4610/5000 [15:51:00<1:11:42, 11.03s/it][2022-12-21 02:13:35,448] [INFO] [timer.py:197:stop] 0/9334, RunningAvgSamplesPerSec=5.880678028343625, CurrSamplesPerSec=5.345741462550639, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4611/5000 [15:51:11<1:11:22, 11.01s/it][2022-12-21 02:13:46,470] [INFO] [timer.py:197:stop] 0/9336, RunningAvgSamplesPerSec=5.880676122343861, CurrSamplesPerSec=5.302391264077579, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4612/5000 [15:51:22<1:11:12, 11.01s/it][2022-12-21 02:13:57,429] [INFO] [timer.py:197:stop] 0/9338, RunningAvgSamplesPerSec=5.880681502507283, CurrSamplesPerSec=5.361325716676022, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4613/5000 [15:51:33<1:10:55, 11.00s/it][2022-12-21 02:14:08,411] [INFO] [logging.py:68:log_dist] [Rank 0] step=4670, skipped=7, lr=[7.511111111111112e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:14:08,412] [INFO] [timer.py:197:stop] 0/9340, RunningAvgSamplesPerSec=5.880684103652808, CurrSamplesPerSec=5.323047380801796, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4614/5000 [15:51:44<1:10:43, 10.99s/it][2022-12-21 02:14:19,364] [INFO] [timer.py:197:stop] 0/9342, RunningAvgSamplesPerSec=5.880690577382234, CurrSamplesPerSec=5.350064146449623, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4615/5000 [15:51:54<1:10:27, 10.98s/it][2022-12-21 02:14:30,383] [INFO] [timer.py:197:stop] 0/9344, RunningAvgSamplesPerSec=5.880689073771709, CurrSamplesPerSec=5.331595033913721, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4616/5000 [15:52:05<1:10:20, 10.99s/it][2022-12-21 02:14:41,366] [INFO] [timer.py:197:stop] 0/9346, RunningAvgSamplesPerSec=5.880691773136352, CurrSamplesPerSec=5.335867707996117, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4617/5000 [15:52:16<1:10:08, 10.99s/it][2022-12-21 02:14:52,313] [INFO] [timer.py:197:stop] 0/9348, RunningAvgSamplesPerSec=5.880698594052202, CurrSamplesPerSec=5.3455264271695935, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4618/5000 [15:52:27<1:09:53, 10.98s/it][2022-12-21 02:15:03,327] [INFO] [timer.py:197:stop] 0/9350, RunningAvgSamplesPerSec=5.880697658563655, CurrSamplesPerSec=5.333835861079956, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4619/5000 [15:52:38<1:09:46, 10.99s/it][2022-12-21 02:15:14,347] [INFO] [timer.py:197:stop] 0/9352, RunningAvgSamplesPerSec=5.8806961643112565, CurrSamplesPerSec=5.337276829296905, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4620/5000 [15:52:49<1:09:39, 11.00s/it][2022-12-21 02:15:25,339] [INFO] [timer.py:197:stop] 0/9354, RunningAvgSamplesPerSec=5.8806977989082085, CurrSamplesPerSec=5.350207673847223, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4621/5000 [15:53:00<1:09:27, 11.00s/it][2022-12-21 02:15:36,307] [INFO] [timer.py:197:stop] 0/9356, RunningAvgSamplesPerSec=5.880702191766011, CurrSamplesPerSec=5.372426194235268, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4622/5000 [15:53:11<1:09:13, 10.99s/it][2022-12-21 02:15:47,291] [INFO] [timer.py:197:stop] 0/9358, RunningAvgSamplesPerSec=5.880704831476866, CurrSamplesPerSec=5.33752749785861, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4623/5000 [15:53:22<1:09:01, 10.99s/it][2022-12-21 02:15:58,294] [INFO] [logging.py:68:log_dist] [Rank 0] step=4680, skipped=7, lr=[7.28888888888889e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:15:58,295] [INFO] [timer.py:197:stop] 0/9360, RunningAvgSamplesPerSec=5.880704982383556, CurrSamplesPerSec=5.3099899764429, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▏| 4624/5000 [15:53:33<1:08:52, 10.99s/it][2022-12-21 02:16:09,277] [INFO] [timer.py:197:stop] 0/9362, RunningAvgSamplesPerSec=5.880707774198584, CurrSamplesPerSec=5.350771621278556, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 92%|█████████▎| 4625/5000 [15:53:44<1:08:40, 10.99s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 7.266666666666668e-07, 'epoch': 112.8}

 92%|█████████▎| 4625/5000 [15:53:44<1:08:40, 10.99s/it][2022-12-21 02:16:20,337] [INFO] [timer.py:197:stop] 0/9364, RunningAvgSamplesPerSec=5.8807017191884965, CurrSamplesPerSec=5.289321899187964, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4626/5000 [15:53:55<1:08:37, 11.01s/it][2022-12-21 02:16:31,342] [INFO] [timer.py:197:stop] 0/9366, RunningAvgSamplesPerSec=5.880701937197023, CurrSamplesPerSec=5.336895883637636, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4627/5000 [15:54:06<1:08:26, 11.01s/it][2022-12-21 02:16:42,364] [INFO] [timer.py:197:stop] 0/9368, RunningAvgSamplesPerSec=5.880700082216833, CurrSamplesPerSec=5.337319490107559, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4628/5000 [15:54:17<1:08:16, 11.01s/it][2022-12-21 02:16:53,368] [INFO] [timer.py:197:stop] 0/9370, RunningAvgSamplesPerSec=5.880700452525664, CurrSamplesPerSec=5.327815366597375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4629/5000 [15:54:28<1:08:04, 11.01s/it][2022-12-21 02:17:04,392] [INFO] [timer.py:197:stop] 0/9372, RunningAvgSamplesPerSec=5.880698355623813, CurrSamplesPerSec=5.3258513550438, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4630/5000 [15:54:39<1:07:55, 11.01s/it][2022-12-21 02:17:15,404] [INFO] [timer.py:197:stop] 0/9374, RunningAvgSamplesPerSec=5.880697516578974, CurrSamplesPerSec=5.323837475269936, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4631/5000 [15:54:50<1:07:44, 11.01s/it][2022-12-21 02:17:26,474] [INFO] [timer.py:197:stop] 0/9376, RunningAvgSamplesPerSec=5.880690092606823, CurrSamplesPerSec=5.282405392001692, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4632/5000 [15:55:02<1:07:39, 11.03s/it][2022-12-21 02:17:37,507] [INFO] [timer.py:197:stop] 0/9378, RunningAvgSamplesPerSec=5.880687165305491, CurrSamplesPerSec=5.306850559413277, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4633/5000 [15:55:13<1:07:28, 11.03s/it][2022-12-21 02:17:47,620] [INFO] [logging.py:68:log_dist] [Rank 0] step=4690, skipped=7, lr=[7.066666666666667e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:17:47,621] [INFO] [timer.py:197:stop] 0/9380, RunningAvgSamplesPerSec=5.88078889299912, CurrSamplesPerSec=5.314390311475187, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4634/5000 [15:55:28<1:14:36, 12.23s/it][2022-12-21 02:17:58,568] [INFO] [timer.py:197:stop] 0/9382, RunningAvgSamplesPerSec=5.880795693869737, CurrSamplesPerSec=5.364394859078534, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4635/5000 [15:55:39<1:12:06, 11.85s/it][2022-12-21 02:18:09,570] [INFO] [timer.py:197:stop] 0/9384, RunningAvgSamplesPerSec=5.880796944054113, CurrSamplesPerSec=5.339768217035852, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4636/5000 [15:55:50<1:10:18, 11.59s/it][2022-12-21 02:18:20,551] [INFO] [timer.py:197:stop] 0/9386, RunningAvgSamplesPerSec=5.880800058239515, CurrSamplesPerSec=5.334349507949397, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4637/5000 [15:56:01<1:08:58, 11.40s/it][2022-12-21 02:18:31,539] [INFO] [timer.py:197:stop] 0/9388, RunningAvgSamplesPerSec=5.880802089305658, CurrSamplesPerSec=5.309938088048904, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4638/5000 [15:56:12<1:08:08, 11.29s/it][2022-12-21 02:18:42,572] [INFO] [timer.py:197:stop] 0/9390, RunningAvgSamplesPerSec=5.88079884255112, CurrSamplesPerSec=5.320466323656873, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4639/5000 [15:56:23<1:07:29, 11.22s/it][2022-12-21 02:18:53,616] [INFO] [timer.py:197:stop] 0/9392, RunningAvgSamplesPerSec=5.88079460272898, CurrSamplesPerSec=5.31288010080947, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4640/5000 [15:56:34<1:06:59, 11.16s/it][2022-12-21 02:19:04,632] [INFO] [timer.py:197:stop] 0/9394, RunningAvgSamplesPerSec=5.880793588239236, CurrSamplesPerSec=5.336838162911671, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4641/5000 [15:56:45<1:06:31, 11.12s/it][2022-12-21 02:19:15,662] [INFO] [timer.py:197:stop] 0/9396, RunningAvgSamplesPerSec=5.880790969306624, CurrSamplesPerSec=5.32137527408203, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4642/5000 [15:56:56<1:06:12, 11.10s/it][2022-12-21 02:19:26,637] [INFO] [timer.py:197:stop] 0/9398, RunningAvgSamplesPerSec=5.880794466351256, CurrSamplesPerSec=5.384895253548928, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4643/5000 [15:57:07<1:05:42, 11.04s/it][2022-12-21 02:19:37,623] [INFO] [logging.py:68:log_dist] [Rank 0] step=4700, skipped=7, lr=[6.844444444444446e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:19:37,624] [INFO] [timer.py:197:stop] 0/9400, RunningAvgSamplesPerSec=5.8807965547931556, CurrSamplesPerSec=5.322559125940257, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4644/5000 [15:57:18<1:05:29, 11.04s/it][2022-12-21 02:19:48,671] [INFO] [timer.py:197:stop] 0/9402, RunningAvgSamplesPerSec=5.880791747535703, CurrSamplesPerSec=5.304128596194462, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4645/5000 [15:57:29<1:05:18, 11.04s/it][2022-12-21 02:19:59,662] [INFO] [timer.py:197:stop] 0/9404, RunningAvgSamplesPerSec=5.880793389198383, CurrSamplesPerSec=5.3488081305172726, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4646/5000 [15:57:40<1:04:58, 11.01s/it][2022-12-21 02:20:10,653] [INFO] [timer.py:197:stop] 0/9406, RunningAvgSamplesPerSec=5.880795151353624, CurrSamplesPerSec=5.311056539057226, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4647/5000 [15:57:51<1:04:49, 11.02s/it][2022-12-21 02:20:21,681] [INFO] [timer.py:197:stop] 0/9408, RunningAvgSamplesPerSec=5.880792506774223, CurrSamplesPerSec=5.318655466154241, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4648/5000 [15:58:02<1:04:37, 11.01s/it][2022-12-21 02:20:32,666] [INFO] [timer.py:197:stop] 0/9410, RunningAvgSamplesPerSec=5.8807949759984215, CurrSamplesPerSec=5.334540958398183, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4649/5000 [15:58:13<1:04:27, 11.02s/it][2022-12-21 02:20:43,712] [INFO] [timer.py:197:stop] 0/9412, RunningAvgSamplesPerSec=5.880790505378182, CurrSamplesPerSec=5.319458171792316, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4650/5000 [15:58:24<1:04:14, 11.01s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 6.711111111111111e-07, 'epoch': 113.41}

 93%|█████████▎| 4650/5000 [15:58:24<1:04:14, 11.01s/it][2022-12-21 02:20:54,678] [INFO] [timer.py:197:stop] 0/9414, RunningAvgSamplesPerSec=5.880795394153776, CurrSamplesPerSec=5.353775512735795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4651/5000 [15:58:35<1:04:02, 11.01s/it][2022-12-21 02:21:05,723] [INFO] [timer.py:197:stop] 0/9416, RunningAvgSamplesPerSec=5.8807909106728795, CurrSamplesPerSec=5.316965686015462, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4652/5000 [15:58:46<1:03:55, 11.02s/it][2022-12-21 02:21:16,746] [INFO] [timer.py:197:stop] 0/9418, RunningAvgSamplesPerSec=5.880788914360871, CurrSamplesPerSec=5.332761190127796, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4653/5000 [15:58:57<1:03:41, 11.01s/it][2022-12-21 02:21:27,765] [INFO] [logging.py:68:log_dist] [Rank 0] step=4710, skipped=7, lr=[6.622222222222223e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:21:27,766] [INFO] [timer.py:197:stop] 0/9420, RunningAvgSamplesPerSec=5.880787360146206, CurrSamplesPerSec=5.309719621724139, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4654/5000 [15:59:08<1:03:31, 11.01s/it][2022-12-21 02:21:38,775] [INFO] [timer.py:197:stop] 0/9422, RunningAvgSamplesPerSec=5.880786969491311, CurrSamplesPerSec=5.319043719459168, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4655/5000 [15:59:19<1:03:17, 11.01s/it][2022-12-21 02:21:49,725] [INFO] [timer.py:197:stop] 0/9424, RunningAvgSamplesPerSec=5.880793473099913, CurrSamplesPerSec=5.356657283910076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4656/5000 [15:59:30<1:03:04, 11.00s/it][2022-12-21 02:22:00,725] [INFO] [timer.py:197:stop] 0/9426, RunningAvgSamplesPerSec=5.880794147833345, CurrSamplesPerSec=5.3453995431203065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4657/5000 [15:59:41<1:02:50, 10.99s/it][2022-12-21 02:22:11,675] [INFO] [timer.py:197:stop] 0/9428, RunningAvgSamplesPerSec=5.880800444833497, CurrSamplesPerSec=5.368231347410081, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4658/5000 [15:59:52<1:02:37, 10.99s/it][2022-12-21 02:22:22,710] [INFO] [timer.py:197:stop] 0/9430, RunningAvgSamplesPerSec=5.880797053989978, CurrSamplesPerSec=5.313987799238561, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4659/5000 [16:00:03<1:02:33, 11.01s/it][2022-12-21 02:22:33,781] [INFO] [timer.py:197:stop] 0/9432, RunningAvgSamplesPerSec=5.880789627469041, CurrSamplesPerSec=5.294869412707697, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4660/5000 [16:00:14<1:02:27, 11.02s/it][2022-12-21 02:22:44,786] [INFO] [timer.py:197:stop] 0/9434, RunningAvgSamplesPerSec=5.880789619836106, CurrSamplesPerSec=5.339970891851903, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4661/5000 [16:00:25<1:02:15, 11.02s/it][2022-12-21 02:22:55,800] [INFO] [timer.py:197:stop] 0/9436, RunningAvgSamplesPerSec=5.880788769089052, CurrSamplesPerSec=5.3382102168041285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4662/5000 [16:00:36<1:02:03, 11.02s/it][2022-12-21 02:23:06,833] [INFO] [timer.py:197:stop] 0/9438, RunningAvgSamplesPerSec=5.880785622079926, CurrSamplesPerSec=5.316435375499112, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4663/5000 [16:00:47<1:01:51, 11.01s/it][2022-12-21 02:23:17,865] [INFO] [logging.py:68:log_dist] [Rank 0] step=4720, skipped=7, lr=[6.4e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:23:17,866] [INFO] [timer.py:197:stop] 0/9440, RunningAvgSamplesPerSec=5.880782435237846, CurrSamplesPerSec=5.2971576570048535, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4664/5000 [16:00:58<1:01:45, 11.03s/it][2022-12-21 02:23:28,886] [INFO] [timer.py:197:stop] 0/9442, RunningAvgSamplesPerSec=5.880780786721044, CurrSamplesPerSec=5.331859148504126, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4665/5000 [16:01:09<1:01:33, 11.02s/it][2022-12-21 02:23:39,922] [INFO] [timer.py:197:stop] 0/9444, RunningAvgSamplesPerSec=5.880777362138165, CurrSamplesPerSec=5.313758901548024, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4666/5000 [16:01:20<1:01:21, 11.02s/it][2022-12-21 02:23:50,886] [INFO] [timer.py:197:stop] 0/9446, RunningAvgSamplesPerSec=5.880782115397386, CurrSamplesPerSec=5.361493407712165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4667/5000 [16:01:31<1:01:02, 11.00s/it][2022-12-21 02:24:01,855] [INFO] [timer.py:197:stop] 0/9448, RunningAvgSamplesPerSec=5.880786423819913, CurrSamplesPerSec=5.338266693293489, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4668/5000 [16:01:42<1:00:46, 10.98s/it][2022-12-21 02:24:12,827] [INFO] [timer.py:197:stop] 0/9450, RunningAvgSamplesPerSec=5.880790260114006, CurrSamplesPerSec=5.313325144028703, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4669/5000 [16:01:53<1:00:42, 11.00s/it][2022-12-21 02:24:23,842] [INFO] [timer.py:197:stop] 0/9452, RunningAvgSamplesPerSec=5.880789191067208, CurrSamplesPerSec=5.350944625955184, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4670/5000 [16:02:04<1:00:28, 11.00s/it][2022-12-21 02:24:34,853] [INFO] [timer.py:197:stop] 0/9454, RunningAvgSamplesPerSec=5.880788568813164, CurrSamplesPerSec=5.317789583049633, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4671/5000 [16:02:15<1:00:20, 11.01s/it][2022-12-21 02:24:45,843] [INFO] [timer.py:197:stop] 0/9456, RunningAvgSamplesPerSec=5.880790376603267, CurrSamplesPerSec=5.353884214609746, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4672/5000 [16:02:26<1:00:09, 11.00s/it][2022-12-21 02:24:56,880] [INFO] [timer.py:197:stop] 0/9458, RunningAvgSamplesPerSec=5.880786852021977, CurrSamplesPerSec=5.322901296026654, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4673/5000 [16:02:37<59:57, 11.00s/it]  [2022-12-21 02:25:07,873] [INFO] [logging.py:68:log_dist] [Rank 0] step=4730, skipped=7, lr=[6.177777777777778e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:25:07,875] [INFO] [timer.py:197:stop] 0/9460, RunningAvgSamplesPerSec=5.880788068581071, CurrSamplesPerSec=5.324713145166745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 93%|█████████▎| 4674/5000 [16:02:48<59:50, 11.02s/it][2022-12-21 02:25:17,997] [INFO] [timer.py:197:stop] 0/9462, RunningAvgSamplesPerSec=5.880887851594186, CurrSamplesPerSec=6.278946082906602, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 02:25:29,068] [INFO] [timer.py:197:stop] 0/9464, RunningAvgSamplesPerSec=5.880880049918166, CurrSamplesPerSec=5.30392276421961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4675/5000 [16:03:04<1:08:07, 12.58s/it]
                                                        
{'loss': 0.0001, 'learning_rate': 6.133333333333333e-07, 'epoch': 114.02}

 94%|█████████▎| 4675/5000 [16:03:04<1:08:07, 12.58s/it][2022-12-21 02:25:40,117] [INFO] [timer.py:197:stop] 0/9466, RunningAvgSamplesPerSec=5.880875437495437, CurrSamplesPerSec=5.313117756390179, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4676/5000 [16:03:15<1:05:26, 12.12s/it][2022-12-21 02:25:51,147] [INFO] [timer.py:197:stop] 0/9468, RunningAvgSamplesPerSec=5.880872643743671, CurrSamplesPerSec=5.30335942805252, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4677/5000 [16:03:26<1:03:28, 11.79s/it][2022-12-21 02:26:02,209] [INFO] [timer.py:197:stop] 0/9470, RunningAvgSamplesPerSec=5.880866206270647, CurrSamplesPerSec=5.29196775543393, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4678/5000 [16:03:37<1:02:06, 11.57s/it][2022-12-21 02:26:13,261] [INFO] [timer.py:197:stop] 0/9472, RunningAvgSamplesPerSec=5.8808610317970444, CurrSamplesPerSec=5.316433901391751, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4679/5000 [16:03:48<1:01:04, 11.42s/it][2022-12-21 02:26:24,285] [INFO] [timer.py:197:stop] 0/9474, RunningAvgSamplesPerSec=5.880858933847421, CurrSamplesPerSec=5.322914173119551, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4680/5000 [16:03:59<1:00:15, 11.30s/it][2022-12-21 02:26:35,277] [INFO] [timer.py:197:stop] 0/9476, RunningAvgSamplesPerSec=5.880860279925671, CurrSamplesPerSec=5.365175829805524, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4681/5000 [16:04:10<59:35, 11.21s/it]  [2022-12-21 02:26:46,287] [INFO] [timer.py:197:stop] 0/9478, RunningAvgSamplesPerSec=5.880859778939343, CurrSamplesPerSec=5.335196828052147, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4682/5000 [16:04:21<59:04, 11.15s/it][2022-12-21 02:26:57,288] [INFO] [logging.py:68:log_dist] [Rank 0] step=4740, skipped=7, lr=[5.955555555555556e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:26:57,290] [INFO] [timer.py:197:stop] 0/9480, RunningAvgSamplesPerSec=5.880860078701431, CurrSamplesPerSec=5.338846177131293, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4683/5000 [16:04:32<58:40, 11.10s/it][2022-12-21 02:27:08,307] [INFO] [timer.py:197:stop] 0/9482, RunningAvgSamplesPerSec=5.88085880385782, CurrSamplesPerSec=5.334523148474391, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4684/5000 [16:04:43<58:20, 11.08s/it][2022-12-21 02:27:19,244] [INFO] [timer.py:197:stop] 0/9484, RunningAvgSamplesPerSec=5.880866586789677, CurrSamplesPerSec=5.369438074913051, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4685/5000 [16:04:54<57:56, 11.04s/it][2022-12-21 02:27:30,233] [INFO] [timer.py:197:stop] 0/9486, RunningAvgSamplesPerSec=5.880868364957599, CurrSamplesPerSec=5.334285906190174, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4686/5000 [16:05:05<57:40, 11.02s/it][2022-12-21 02:27:41,264] [INFO] [timer.py:197:stop] 0/9488, RunningAvgSamplesPerSec=5.880865537576707, CurrSamplesPerSec=5.322996081332886, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▎| 4687/5000 [16:05:16<57:30, 11.02s/it][2022-12-21 02:27:52,269] [INFO] [timer.py:197:stop] 0/9490, RunningAvgSamplesPerSec=5.880865503259718, CurrSamplesPerSec=5.339914166846696, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4688/5000 [16:05:27<57:17, 11.02s/it][2022-12-21 02:28:03,211] [INFO] [timer.py:197:stop] 0/9492, RunningAvgSamplesPerSec=5.88087274333504, CurrSamplesPerSec=5.37592856198446, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4689/5000 [16:05:38<56:59, 11.00s/it][2022-12-21 02:28:14,206] [INFO] [timer.py:197:stop] 0/9494, RunningAvgSamplesPerSec=5.880874334602242, CurrSamplesPerSec=5.311362971361763, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4690/5000 [16:05:49<56:48, 11.00s/it][2022-12-21 02:28:25,251] [INFO] [timer.py:197:stop] 0/9496, RunningAvgSamplesPerSec=5.88086991590282, CurrSamplesPerSec=5.307259336211239, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4691/5000 [16:06:00<56:42, 11.01s/it][2022-12-21 02:28:36,314] [INFO] [timer.py:197:stop] 0/9498, RunningAvgSamplesPerSec=5.880863501356603, CurrSamplesPerSec=5.312874212268554, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4692/5000 [16:06:11<56:35, 11.03s/it][2022-12-21 02:28:47,290] [INFO] [logging.py:68:log_dist] [Rank 0] step=4750, skipped=7, lr=[5.733333333333334e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:28:47,292] [INFO] [timer.py:197:stop] 0/9500, RunningAvgSamplesPerSec=5.880866705535532, CurrSamplesPerSec=5.343637627235034, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4693/5000 [16:06:22<56:20, 11.01s/it][2022-12-21 02:28:58,233] [INFO] [timer.py:197:stop] 0/9502, RunningAvgSamplesPerSec=5.880874005657511, CurrSamplesPerSec=5.372548988212856, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4694/5000 [16:06:33<56:03, 10.99s/it][2022-12-21 02:29:09,222] [INFO] [timer.py:197:stop] 0/9504, RunningAvgSamplesPerSec=5.8808759740417, CurrSamplesPerSec=5.319229856642703, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4695/5000 [16:06:44<55:51, 10.99s/it][2022-12-21 02:29:20,217] [INFO] [timer.py:197:stop] 0/9506, RunningAvgSamplesPerSec=5.880877130395758, CurrSamplesPerSec=5.344237642067467, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4696/5000 [16:06:55<55:41, 10.99s/it][2022-12-21 02:29:31,221] [INFO] [timer.py:197:stop] 0/9508, RunningAvgSamplesPerSec=5.880877367509262, CurrSamplesPerSec=5.321058403469919, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4697/5000 [16:07:06<55:31, 11.00s/it][2022-12-21 02:29:42,247] [INFO] [timer.py:197:stop] 0/9510, RunningAvgSamplesPerSec=5.880875019658428, CurrSamplesPerSec=5.2961263439907675, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4698/5000 [16:07:17<55:23, 11.00s/it][2022-12-21 02:29:53,202] [INFO] [timer.py:197:stop] 0/9512, RunningAvgSamplesPerSec=5.8808810340777145, CurrSamplesPerSec=5.36053344880522, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4699/5000 [16:07:28<55:07, 10.99s/it][2022-12-21 02:30:04,236] [INFO] [timer.py:197:stop] 0/9514, RunningAvgSamplesPerSec=5.880877955427366, CurrSamplesPerSec=5.307913552073799, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4700/5000 [16:07:39<55:00, 11.00s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 5.577777777777779e-07, 'epoch': 114.63}

 94%|█████████▍| 4700/5000 [16:07:39<55:00, 11.00s/it][2022-12-21 02:30:15,204] [INFO] [timer.py:197:stop] 0/9516, RunningAvgSamplesPerSec=5.880882361982548, CurrSamplesPerSec=5.362196624955059, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4701/5000 [16:07:50<54:46, 10.99s/it][2022-12-21 02:30:26,223] [INFO] [timer.py:197:stop] 0/9518, RunningAvgSamplesPerSec=5.880880979605738, CurrSamplesPerSec=5.298048203423232, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4702/5000 [16:08:01<54:38, 11.00s/it][2022-12-21 02:30:37,169] [INFO] [logging.py:68:log_dist] [Rank 0] step=4760, skipped=7, lr=[5.511111111111111e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:30:37,170] [INFO] [timer.py:197:stop] 0/9520, RunningAvgSamplesPerSec=5.880887499212462, CurrSamplesPerSec=5.359891455368944, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4703/5000 [16:08:12<54:22, 10.98s/it][2022-12-21 02:30:48,166] [INFO] [timer.py:197:stop] 0/9522, RunningAvgSamplesPerSec=5.8808886008077, CurrSamplesPerSec=5.357850685486765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4704/5000 [16:08:23<54:12, 10.99s/it][2022-12-21 02:30:59,214] [INFO] [timer.py:197:stop] 0/9524, RunningAvgSamplesPerSec=5.8808838104937795, CurrSamplesPerSec=5.3304448415317545, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4705/5000 [16:08:34<54:06, 11.01s/it][2022-12-21 02:31:10,238] [INFO] [timer.py:197:stop] 0/9526, RunningAvgSamplesPerSec=5.880881720010783, CurrSamplesPerSec=5.3113793658559985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4706/5000 [16:08:45<53:57, 11.01s/it][2022-12-21 02:31:21,184] [INFO] [timer.py:197:stop] 0/9528, RunningAvgSamplesPerSec=5.880888385630128, CurrSamplesPerSec=5.38144916274789, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4707/5000 [16:08:56<53:40, 10.99s/it][2022-12-21 02:31:32,139] [INFO] [timer.py:197:stop] 0/9530, RunningAvgSamplesPerSec=5.880894165173551, CurrSamplesPerSec=5.3629488838628205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4708/5000 [16:09:07<53:26, 10.98s/it][2022-12-21 02:31:43,121] [INFO] [timer.py:197:stop] 0/9532, RunningAvgSamplesPerSec=5.880896884017917, CurrSamplesPerSec=5.328187825333314, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4709/5000 [16:09:18<53:15, 10.98s/it][2022-12-21 02:31:54,148] [INFO] [timer.py:197:stop] 0/9534, RunningAvgSamplesPerSec=5.880894517936693, CurrSamplesPerSec=5.31600897138238, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4710/5000 [16:09:29<53:08, 10.99s/it][2022-12-21 02:32:05,118] [INFO] [timer.py:197:stop] 0/9536, RunningAvgSamplesPerSec=5.880898635147077, CurrSamplesPerSec=5.345557510422361, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4711/5000 [16:09:40<52:55, 10.99s/it][2022-12-21 02:32:16,078] [INFO] [timer.py:197:stop] 0/9538, RunningAvgSamplesPerSec=5.880903868224854, CurrSamplesPerSec=5.338669919192283, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4712/5000 [16:09:51<52:41, 10.98s/it][2022-12-21 02:32:27,076] [INFO] [logging.py:68:log_dist] [Rank 0] step=4770, skipped=7, lr=[5.288888888888889e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:32:27,077] [INFO] [timer.py:197:stop] 0/9540, RunningAvgSamplesPerSec=5.880904764747687, CurrSamplesPerSec=5.360762111814961, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4713/5000 [16:10:02<52:32, 10.99s/it][2022-12-21 02:32:38,107] [INFO] [timer.py:197:stop] 0/9542, RunningAvgSamplesPerSec=5.8809021717543235, CurrSamplesPerSec=5.3177225831637225, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4714/5000 [16:10:13<52:25, 11.00s/it][2022-12-21 02:32:49,151] [INFO] [timer.py:197:stop] 0/9544, RunningAvgSamplesPerSec=5.880897999404677, CurrSamplesPerSec=5.321691127418676, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4715/5000 [16:10:24<52:18, 11.01s/it][2022-12-21 02:32:59,242] [INFO] [timer.py:197:stop] 0/9546, RunningAvgSamplesPerSec=5.881000593704332, CurrSamplesPerSec=5.34906180203864, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4716/5000 [16:10:39<57:49, 12.22s/it][2022-12-21 02:33:10,275] [INFO] [timer.py:197:stop] 0/9548, RunningAvgSamplesPerSec=5.880997517581248, CurrSamplesPerSec=5.3081931699300755, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4717/5000 [16:10:50<55:57, 11.86s/it][2022-12-21 02:33:21,271] [INFO] [timer.py:197:stop] 0/9550, RunningAvgSamplesPerSec=5.88099863134607, CurrSamplesPerSec=5.344514077211338, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4718/5000 [16:11:01<54:29, 11.59s/it][2022-12-21 02:33:32,279] [INFO] [timer.py:197:stop] 0/9552, RunningAvgSamplesPerSec=5.880998249494198, CurrSamplesPerSec=5.30742660031388, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4719/5000 [16:11:12<53:31, 11.43s/it][2022-12-21 02:33:43,317] [INFO] [timer.py:197:stop] 0/9554, RunningAvgSamplesPerSec=5.880994565301846, CurrSamplesPerSec=5.309396366558196, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4720/5000 [16:11:23<52:45, 11.31s/it][2022-12-21 02:33:54,324] [INFO] [timer.py:197:stop] 0/9556, RunningAvgSamplesPerSec=5.880994213833595, CurrSamplesPerSec=5.31832964692065, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4721/5000 [16:11:34<52:12, 11.23s/it][2022-12-21 02:34:05,387] [INFO] [timer.py:197:stop] 0/9558, RunningAvgSamplesPerSec=5.8809884030759605, CurrSamplesPerSec=5.306666336658286, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4722/5000 [16:11:45<51:43, 11.16s/it][2022-12-21 02:34:16,370] [INFO] [logging.py:68:log_dist] [Rank 0] step=4780, skipped=7, lr=[5.066666666666667e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:34:16,371] [INFO] [timer.py:197:stop] 0/9560, RunningAvgSamplesPerSec=5.880990807735305, CurrSamplesPerSec=5.33344798864372, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4723/5000 [16:11:56<51:20, 11.12s/it][2022-12-21 02:34:27,348] [INFO] [timer.py:197:stop] 0/9562, RunningAvgSamplesPerSec=5.880994155099633, CurrSamplesPerSec=5.36987567179094, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4724/5000 [16:12:07<50:56, 11.08s/it][2022-12-21 02:34:38,374] [INFO] [timer.py:197:stop] 0/9564, RunningAvgSamplesPerSec=5.88099197358117, CurrSamplesPerSec=5.320407903258427, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 94%|█████████▍| 4725/5000 [16:12:18<50:37, 11.05s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 5.022222222222222e-07, 'epoch': 115.24}

 94%|█████████▍| 4725/5000 [16:12:18<50:37, 11.05s/it][2022-12-21 02:34:49,358] [INFO] [timer.py:197:stop] 0/9566, RunningAvgSamplesPerSec=5.880994530831817, CurrSamplesPerSec=5.31895518765823, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4726/5000 [16:12:29<50:24, 11.04s/it][2022-12-21 02:35:00,331] [INFO] [timer.py:197:stop] 0/9568, RunningAvgSamplesPerSec=5.880998263657485, CurrSamplesPerSec=5.35284862951076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4727/5000 [16:12:40<50:09, 11.02s/it][2022-12-21 02:35:11,355] [INFO] [timer.py:197:stop] 0/9570, RunningAvgSamplesPerSec=5.880996240494401, CurrSamplesPerSec=5.321895386545972, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4728/5000 [16:12:51<49:59, 11.03s/it][2022-12-21 02:35:22,391] [INFO] [timer.py:197:stop] 0/9572, RunningAvgSamplesPerSec=5.880992826698634, CurrSamplesPerSec=5.3294051831268545, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4729/5000 [16:13:02<49:45, 11.02s/it][2022-12-21 02:35:33,402] [INFO] [timer.py:197:stop] 0/9574, RunningAvgSamplesPerSec=5.880992126929935, CurrSamplesPerSec=5.305552040729241, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4730/5000 [16:13:13<49:34, 11.02s/it][2022-12-21 02:35:44,399] [INFO] [timer.py:197:stop] 0/9576, RunningAvgSamplesPerSec=5.880993230611954, CurrSamplesPerSec=5.328104699704492, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4731/5000 [16:13:24<49:20, 11.01s/it][2022-12-21 02:35:55,398] [INFO] [timer.py:197:stop] 0/9578, RunningAvgSamplesPerSec=5.8809940315845735, CurrSamplesPerSec=5.31392825869537, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4732/5000 [16:13:35<49:13, 11.02s/it][2022-12-21 02:36:06,442] [INFO] [logging.py:68:log_dist] [Rank 0] step=4790, skipped=7, lr=[4.844444444444445e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:36:06,443] [INFO] [timer.py:197:stop] 0/9580, RunningAvgSamplesPerSec=5.880989476905555, CurrSamplesPerSec=5.315551689506329, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4733/5000 [16:13:47<49:04, 11.03s/it][2022-12-21 02:36:17,499] [INFO] [timer.py:197:stop] 0/9582, RunningAvgSamplesPerSec=5.880983864531018, CurrSamplesPerSec=5.3123916065958205, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4734/5000 [16:13:58<48:55, 11.03s/it][2022-12-21 02:36:28,515] [INFO] [timer.py:197:stop] 0/9584, RunningAvgSamplesPerSec=5.880982646508299, CurrSamplesPerSec=5.340226488003584, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4735/5000 [16:14:09<48:44, 11.03s/it][2022-12-21 02:36:39,537] [INFO] [timer.py:197:stop] 0/9586, RunningAvgSamplesPerSec=5.880980679303313, CurrSamplesPerSec=5.351560365901092, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4736/5000 [16:14:20<48:31, 11.03s/it][2022-12-21 02:36:50,545] [INFO] [timer.py:197:stop] 0/9588, RunningAvgSamplesPerSec=5.880980302633258, CurrSamplesPerSec=5.351210234436573, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4737/5000 [16:14:31<48:15, 11.01s/it][2022-12-21 02:37:01,548] [INFO] [timer.py:197:stop] 0/9590, RunningAvgSamplesPerSec=5.880980703100028, CurrSamplesPerSec=5.317763246415105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4738/5000 [16:14:42<48:04, 11.01s/it][2022-12-21 02:37:12,551] [INFO] [timer.py:197:stop] 0/9592, RunningAvgSamplesPerSec=5.880980879302264, CurrSamplesPerSec=5.325121509411445, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4739/5000 [16:14:53<47:55, 11.02s/it][2022-12-21 02:37:23,619] [INFO] [timer.py:197:stop] 0/9594, RunningAvgSamplesPerSec=5.88097407738425, CurrSamplesPerSec=5.298682788010215, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4740/5000 [16:15:04<47:47, 11.03s/it][2022-12-21 02:37:34,652] [INFO] [timer.py:197:stop] 0/9596, RunningAvgSamplesPerSec=5.880970959813388, CurrSamplesPerSec=5.323124015183574, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4741/5000 [16:15:15<47:36, 11.03s/it][2022-12-21 02:37:45,720] [INFO] [timer.py:197:stop] 0/9598, RunningAvgSamplesPerSec=5.880964069559679, CurrSamplesPerSec=5.28422430766218, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4742/5000 [16:15:26<47:30, 11.05s/it][2022-12-21 02:37:56,752] [INFO] [logging.py:68:log_dist] [Rank 0] step=4800, skipped=7, lr=[4.6222222222222225e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:37:56,754] [INFO] [timer.py:197:stop] 0/9600, RunningAvgSamplesPerSec=5.880960865310276, CurrSamplesPerSec=5.332412454901974, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4743/5000 [16:15:37<47:15, 11.03s/it][2022-12-21 02:38:07,806] [INFO] [timer.py:197:stop] 0/9602, RunningAvgSamplesPerSec=5.880955429601621, CurrSamplesPerSec=5.282136591548234, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4744/5000 [16:15:48<47:09, 11.05s/it][2022-12-21 02:38:18,873] [INFO] [timer.py:197:stop] 0/9604, RunningAvgSamplesPerSec=5.880948509380528, CurrSamplesPerSec=5.31414665067882, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4745/5000 [16:15:59<46:59, 11.06s/it][2022-12-21 02:38:30,007] [INFO] [timer.py:197:stop] 0/9606, RunningAvgSamplesPerSec=5.880933997160497, CurrSamplesPerSec=5.2495635556628795, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4746/5000 [16:16:10<46:53, 11.08s/it][2022-12-21 02:38:41,082] [INFO] [timer.py:197:stop] 0/9608, RunningAvgSamplesPerSec=5.880926249353229, CurrSamplesPerSec=5.297937783663247, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4747/5000 [16:16:21<46:40, 11.07s/it][2022-12-21 02:38:52,096] [INFO] [timer.py:197:stop] 0/9610, RunningAvgSamplesPerSec=5.88092535747982, CurrSamplesPerSec=5.333825898623005, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4748/5000 [16:16:32<46:25, 11.05s/it][2022-12-21 02:39:03,117] [INFO] [timer.py:197:stop] 0/9612, RunningAvgSamplesPerSec=5.8809235396962585, CurrSamplesPerSec=5.330626484717996, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▍| 4749/5000 [16:16:43<46:06, 11.02s/it][2022-12-21 02:39:14,047] [INFO] [timer.py:197:stop] 0/9614, RunningAvgSamplesPerSec=5.880932031233577, CurrSamplesPerSec=5.348564500607492, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4750/5000 [16:16:54<45:52, 11.01s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 4.466666666666667e-07, 'epoch': 115.84}

 95%|█████████▌| 4750/5000 [16:16:54<45:52, 11.01s/it][2022-12-21 02:39:25,037] [INFO] [timer.py:197:stop] 0/9616, RunningAvgSamplesPerSec=5.880933872372217, CurrSamplesPerSec=5.342722119610216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4751/5000 [16:17:05<45:40, 11.01s/it][2022-12-21 02:39:36,049] [INFO] [timer.py:197:stop] 0/9618, RunningAvgSamplesPerSec=5.880933187277095, CurrSamplesPerSec=5.32421444689869, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4752/5000 [16:17:16<45:30, 11.01s/it][2022-12-21 02:39:47,021] [INFO] [logging.py:68:log_dist] [Rank 0] step=4810, skipped=7, lr=[4.4e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:39:47,022] [INFO] [timer.py:197:stop] 0/9620, RunningAvgSamplesPerSec=5.880936714529368, CurrSamplesPerSec=5.371673639248028, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4753/5000 [16:17:27<45:18, 11.01s/it][2022-12-21 02:39:58,084] [INFO] [timer.py:197:stop] 0/9622, RunningAvgSamplesPerSec=5.8809304312059325, CurrSamplesPerSec=5.308413610177076, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4754/5000 [16:17:38<45:07, 11.01s/it][2022-12-21 02:40:09,090] [INFO] [timer.py:197:stop] 0/9624, RunningAvgSamplesPerSec=5.88093039306304, CurrSamplesPerSec=5.3145104667772936, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4755/5000 [16:17:49<45:02, 11.03s/it][2022-12-21 02:40:20,146] [INFO] [timer.py:197:stop] 0/9626, RunningAvgSamplesPerSec=5.88092492155338, CurrSamplesPerSec=5.341299918987719, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4756/5000 [16:18:00<44:50, 11.03s/it][2022-12-21 02:40:30,303] [INFO] [timer.py:197:stop] 0/9628, RunningAvgSamplesPerSec=5.8810191306122475, CurrSamplesPerSec=6.225931476021153, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 02:40:41,345] [INFO] [timer.py:197:stop] 0/9630, RunningAvgSamplesPerSec=5.881014691462303, CurrSamplesPerSec=5.318506250321466, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4757/5000 [16:18:16<50:59, 12.59s/it][2022-12-21 02:40:52,392] [INFO] [timer.py:197:stop] 0/9632, RunningAvgSamplesPerSec=5.881010105651238, CurrSamplesPerSec=5.307179800267617, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4758/5000 [16:18:27<48:54, 12.13s/it][2022-12-21 02:41:03,404] [INFO] [timer.py:197:stop] 0/9634, RunningAvgSamplesPerSec=5.881009428453584, CurrSamplesPerSec=5.3380386714875545, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4759/5000 [16:18:38<47:22, 11.79s/it][2022-12-21 02:41:14,426] [INFO] [timer.py:197:stop] 0/9636, RunningAvgSamplesPerSec=5.881007540346251, CurrSamplesPerSec=5.307838404549553, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4760/5000 [16:18:50<46:14, 11.56s/it][2022-12-21 02:41:25,357] [INFO] [timer.py:197:stop] 0/9638, RunningAvgSamplesPerSec=5.881015902804455, CurrSamplesPerSec=5.390849121708894, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4761/5000 [16:19:00<45:17, 11.37s/it][2022-12-21 02:41:36,369] [INFO] [logging.py:68:log_dist] [Rank 0] step=4820, skipped=7, lr=[4.177777777777778e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:41:36,370] [INFO] [timer.py:197:stop] 0/9640, RunningAvgSamplesPerSec=5.881014969675752, CurrSamplesPerSec=5.3340372378693806, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4762/5000 [16:19:11<44:40, 11.26s/it][2022-12-21 02:41:47,445] [INFO] [timer.py:197:stop] 0/9642, RunningAvgSamplesPerSec=5.881007372975275, CurrSamplesPerSec=5.283105066686653, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4763/5000 [16:19:23<44:16, 11.21s/it][2022-12-21 02:41:58,488] [INFO] [timer.py:197:stop] 0/9644, RunningAvgSamplesPerSec=5.881003312608336, CurrSamplesPerSec=5.291778305352447, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4764/5000 [16:19:34<43:53, 11.16s/it][2022-12-21 02:42:09,546] [INFO] [timer.py:197:stop] 0/9646, RunningAvgSamplesPerSec=5.8809974938904555, CurrSamplesPerSec=5.3024880436194115, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4765/5000 [16:19:45<43:35, 11.13s/it][2022-12-21 02:42:20,572] [INFO] [timer.py:197:stop] 0/9648, RunningAvgSamplesPerSec=5.880995268643816, CurrSamplesPerSec=5.339941998065701, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4766/5000 [16:19:56<43:16, 11.10s/it][2022-12-21 02:42:31,609] [INFO] [timer.py:197:stop] 0/9650, RunningAvgSamplesPerSec=5.88099181959029, CurrSamplesPerSec=5.311960805281877, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4767/5000 [16:20:07<43:01, 11.08s/it][2022-12-21 02:42:42,592] [INFO] [timer.py:197:stop] 0/9652, RunningAvgSamplesPerSec=5.880994401294073, CurrSamplesPerSec=5.3276571772799, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4768/5000 [16:20:18<42:43, 11.05s/it][2022-12-21 02:42:53,568] [INFO] [timer.py:197:stop] 0/9654, RunningAvgSamplesPerSec=5.8809977978670585, CurrSamplesPerSec=5.329333234807956, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4769/5000 [16:20:29<42:27, 11.03s/it][2022-12-21 02:43:04,583] [INFO] [timer.py:197:stop] 0/9656, RunningAvgSamplesPerSec=5.880996779421266, CurrSamplesPerSec=5.314460173441543, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4770/5000 [16:20:40<42:15, 11.02s/it][2022-12-21 02:43:15,588] [INFO] [timer.py:197:stop] 0/9658, RunningAvgSamplesPerSec=5.880996899718391, CurrSamplesPerSec=5.303342244827158, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4771/5000 [16:20:51<42:03, 11.02s/it][2022-12-21 02:43:26,602] [INFO] [logging.py:68:log_dist] [Rank 0] step=4830, skipped=7, lr=[3.9555555555555557e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:43:26,603] [INFO] [timer.py:197:stop] 0/9660, RunningAvgSamplesPerSec=5.880995855412996, CurrSamplesPerSec=5.327522681452413, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4772/5000 [16:21:02<41:51, 11.02s/it][2022-12-21 02:43:37,640] [INFO] [timer.py:197:stop] 0/9662, RunningAvgSamplesPerSec=5.880992495430277, CurrSamplesPerSec=5.309967078200697, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4773/5000 [16:21:13<41:42, 11.02s/it][2022-12-21 02:43:48,682] [INFO] [timer.py:197:stop] 0/9664, RunningAvgSamplesPerSec=5.880988487053365, CurrSamplesPerSec=5.320686097079864, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 95%|█████████▌| 4774/5000 [16:21:24<41:32, 11.03s/it][2022-12-21 02:43:59,717] [INFO] [timer.py:197:stop] 0/9666, RunningAvgSamplesPerSec=5.880985200148974, CurrSamplesPerSec=5.318403827350166, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4775/5000 [16:21:35<41:21, 11.03s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 3.8888888888888895e-07, 'epoch': 116.46}

 96%|█████████▌| 4775/5000 [16:21:35<41:21, 11.03s/it][2022-12-21 02:44:10,749] [INFO] [timer.py:197:stop] 0/9668, RunningAvgSamplesPerSec=5.880982451146491, CurrSamplesPerSec=5.33179878319952, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4776/5000 [16:21:46<41:10, 11.03s/it][2022-12-21 02:44:21,728] [INFO] [timer.py:197:stop] 0/9670, RunningAvgSamplesPerSec=5.880985514666687, CurrSamplesPerSec=5.358163183222781, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4777/5000 [16:21:57<40:56, 11.02s/it][2022-12-21 02:44:32,745] [INFO] [timer.py:197:stop] 0/9672, RunningAvgSamplesPerSec=5.880984377710305, CurrSamplesPerSec=5.344094860364544, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4778/5000 [16:22:08<40:45, 11.02s/it][2022-12-21 02:44:43,754] [INFO] [timer.py:197:stop] 0/9674, RunningAvgSamplesPerSec=5.880984125380136, CurrSamplesPerSec=5.309103811508198, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4779/5000 [16:22:19<40:34, 11.01s/it][2022-12-21 02:44:54,779] [INFO] [timer.py:197:stop] 0/9676, RunningAvgSamplesPerSec=5.880982011473916, CurrSamplesPerSec=5.3339658003627575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4780/5000 [16:22:30<40:23, 11.02s/it][2022-12-21 02:45:05,783] [INFO] [timer.py:197:stop] 0/9678, RunningAvgSamplesPerSec=5.8809822052806355, CurrSamplesPerSec=5.342827182847762, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4781/5000 [16:22:41<40:11, 11.01s/it][2022-12-21 02:45:16,787] [INFO] [logging.py:68:log_dist] [Rank 0] step=4840, skipped=7, lr=[3.733333333333334e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:45:16,789] [INFO] [timer.py:197:stop] 0/9680, RunningAvgSamplesPerSec=5.880982075848762, CurrSamplesPerSec=5.311953026691017, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4782/5000 [16:22:52<40:00, 11.01s/it][2022-12-21 02:45:27,777] [INFO] [timer.py:197:stop] 0/9682, RunningAvgSamplesPerSec=5.8809840399254005, CurrSamplesPerSec=5.345632026541421, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4783/5000 [16:23:03<39:47, 11.00s/it][2022-12-21 02:45:38,816] [INFO] [timer.py:197:stop] 0/9684, RunningAvgSamplesPerSec=5.880980450178291, CurrSamplesPerSec=5.307370984354458, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4784/5000 [16:23:14<39:39, 11.02s/it][2022-12-21 02:45:49,780] [INFO] [timer.py:197:stop] 0/9686, RunningAvgSamplesPerSec=5.880985767389646, CurrSamplesPerSec=5.357649217567191, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4785/5000 [16:23:25<39:24, 11.00s/it][2022-12-21 02:46:00,802] [INFO] [timer.py:197:stop] 0/9688, RunningAvgSamplesPerSec=5.880984042902357, CurrSamplesPerSec=5.319238710604539, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4786/5000 [16:23:36<39:15, 11.01s/it][2022-12-21 02:46:11,840] [INFO] [timer.py:197:stop] 0/9690, RunningAvgSamplesPerSec=5.880980420294556, CurrSamplesPerSec=5.282791905564658, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4787/5000 [16:23:47<39:06, 11.02s/it][2022-12-21 02:46:22,768] [INFO] [timer.py:197:stop] 0/9692, RunningAvgSamplesPerSec=5.880989153570312, CurrSamplesPerSec=5.360941116322449, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4788/5000 [16:23:58<38:49, 10.99s/it][2022-12-21 02:46:33,766] [INFO] [timer.py:197:stop] 0/9694, RunningAvgSamplesPerSec=5.8809899894532895, CurrSamplesPerSec=5.374901005958865, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4789/5000 [16:24:09<38:39, 10.99s/it][2022-12-21 02:46:44,791] [INFO] [timer.py:197:stop] 0/9696, RunningAvgSamplesPerSec=5.880987934330421, CurrSamplesPerSec=5.337668442936945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4790/5000 [16:24:20<38:30, 11.00s/it][2022-12-21 02:46:55,745] [INFO] [timer.py:197:stop] 0/9698, RunningAvgSamplesPerSec=5.8809936073969125, CurrSamplesPerSec=5.345198158993703, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4791/5000 [16:24:31<38:16, 10.99s/it][2022-12-21 02:47:06,722] [INFO] [logging.py:68:log_dist] [Rank 0] step=4850, skipped=7, lr=[3.5111111111111117e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:47:06,723] [INFO] [timer.py:197:stop] 0/9700, RunningAvgSamplesPerSec=5.88099698092743, CurrSamplesPerSec=5.324915523756282, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4792/5000 [16:24:42<38:04, 10.98s/it][2022-12-21 02:47:17,708] [INFO] [timer.py:197:stop] 0/9702, RunningAvgSamplesPerSec=5.880999375427564, CurrSamplesPerSec=5.317700882304791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4793/5000 [16:24:53<37:53, 10.98s/it][2022-12-21 02:47:28,669] [INFO] [timer.py:197:stop] 0/9704, RunningAvgSamplesPerSec=5.881004359227824, CurrSamplesPerSec=5.339721693227279, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4794/5000 [16:25:04<37:41, 10.98s/it][2022-12-21 02:47:39,696] [INFO] [timer.py:197:stop] 0/9706, RunningAvgSamplesPerSec=5.881002035340364, CurrSamplesPerSec=5.314884645409777, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4795/5000 [16:25:15<37:33, 10.99s/it][2022-12-21 02:47:50,639] [INFO] [timer.py:197:stop] 0/9708, RunningAvgSamplesPerSec=5.881008963855953, CurrSamplesPerSec=5.35444466523403, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4796/5000 [16:25:26<37:19, 10.98s/it][2022-12-21 02:48:01,688] [INFO] [timer.py:197:stop] 0/9710, RunningAvgSamplesPerSec=5.881005248041847, CurrSamplesPerSec=5.306625842938296, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4797/5000 [16:25:37<37:12, 11.00s/it][2022-12-21 02:48:11,841] [INFO] [timer.py:197:stop] 0/9712, RunningAvgSamplesPerSec=5.881099024491932, CurrSamplesPerSec=5.2881040309257275, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4798/5000 [16:25:52<41:10, 12.23s/it][2022-12-21 02:48:22,904] [INFO] [timer.py:197:stop] 0/9714, RunningAvgSamplesPerSec=5.881092669966116, CurrSamplesPerSec=5.294900118511985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4799/5000 [16:26:03<39:48, 11.88s/it][2022-12-21 02:48:33,916] [INFO] [timer.py:197:stop] 0/9716, RunningAvgSamplesPerSec=5.88109186185115, CurrSamplesPerSec=5.339365674848051, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4800/5000 [16:26:14<38:42, 11.61s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 3.3333333333333335e-07, 'epoch': 117.07}

 96%|█████████▌| 4800/5000 [16:26:14<38:42, 11.61s/it][2022-12-21 02:48:44,927] [INFO] [timer.py:197:stop] 0/9718, RunningAvgSamplesPerSec=5.881091394249807, CurrSamplesPerSec=5.31339602972306, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4801/5000 [16:26:25<37:54, 11.43s/it][2022-12-21 02:48:55,912] [INFO] [logging.py:68:log_dist] [Rank 0] step=4860, skipped=7, lr=[3.2888888888888894e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:48:55,914] [INFO] [timer.py:197:stop] 0/9720, RunningAvgSamplesPerSec=5.881093360889865, CurrSamplesPerSec=5.334486892925552, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4802/5000 [16:26:36<37:18, 11.30s/it][2022-12-21 02:49:06,925] [INFO] [timer.py:197:stop] 0/9722, RunningAvgSamplesPerSec=5.881092555464422, CurrSamplesPerSec=5.332214166512985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4803/5000 [16:26:47<36:48, 11.21s/it][2022-12-21 02:49:17,910] [INFO] [timer.py:197:stop] 0/9724, RunningAvgSamplesPerSec=5.8810947419350965, CurrSamplesPerSec=5.334939168461207, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4804/5000 [16:26:58<36:25, 11.15s/it][2022-12-21 02:49:28,878] [INFO] [timer.py:197:stop] 0/9726, RunningAvgSamplesPerSec=5.881098769008264, CurrSamplesPerSec=5.372962357001803, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4805/5000 [16:27:09<36:04, 11.10s/it][2022-12-21 02:49:39,923] [INFO] [timer.py:197:stop] 0/9728, RunningAvgSamplesPerSec=5.881094302424883, CurrSamplesPerSec=5.316688512914242, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4806/5000 [16:27:20<35:46, 11.07s/it][2022-12-21 02:49:50,932] [INFO] [timer.py:197:stop] 0/9730, RunningAvgSamplesPerSec=5.881093674606915, CurrSamplesPerSec=5.2980216437045, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4807/5000 [16:27:31<35:36, 11.07s/it][2022-12-21 02:50:01,925] [INFO] [timer.py:197:stop] 0/9732, RunningAvgSamplesPerSec=5.8810950256856405, CurrSamplesPerSec=5.373086035884832, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4808/5000 [16:27:42<35:17, 11.03s/it][2022-12-21 02:50:12,883] [INFO] [timer.py:197:stop] 0/9734, RunningAvgSamplesPerSec=5.881100060525745, CurrSamplesPerSec=5.344942725273218, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4809/5000 [16:27:53<35:06, 11.03s/it][2022-12-21 02:50:23,939] [INFO] [timer.py:197:stop] 0/9736, RunningAvgSamplesPerSec=5.881094274824236, CurrSamplesPerSec=5.3198745864512285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4810/5000 [16:28:04<34:54, 11.03s/it][2022-12-21 02:50:34,944] [INFO] [timer.py:197:stop] 0/9738, RunningAvgSamplesPerSec=5.881094227024425, CurrSamplesPerSec=5.337351539224869, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4811/5000 [16:28:15<34:40, 11.01s/it][2022-12-21 02:50:45,928] [INFO] [logging.py:68:log_dist] [Rank 0] step=4870, skipped=7, lr=[3.0666666666666666e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:50:45,930] [INFO] [timer.py:197:stop] 0/9740, RunningAvgSamplesPerSec=5.8810960783248305, CurrSamplesPerSec=5.32104933249672, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▌| 4812/5000 [16:28:26<34:31, 11.02s/it][2022-12-21 02:50:56,937] [INFO] [timer.py:197:stop] 0/9742, RunningAvgSamplesPerSec=5.881095749778431, CurrSamplesPerSec=5.347296832348291, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4813/5000 [16:28:37<34:17, 11.00s/it][2022-12-21 02:51:07,912] [INFO] [timer.py:197:stop] 0/9744, RunningAvgSamplesPerSec=5.881098955435075, CurrSamplesPerSec=5.337720237878165, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4814/5000 [16:28:48<34:04, 10.99s/it][2022-12-21 02:51:18,921] [INFO] [timer.py:197:stop] 0/9746, RunningAvgSamplesPerSec=5.881098458629044, CurrSamplesPerSec=5.3082834432461645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4815/5000 [16:28:59<33:56, 11.01s/it][2022-12-21 02:51:29,949] [INFO] [timer.py:197:stop] 0/9748, RunningAvgSamplesPerSec=5.8810958277765, CurrSamplesPerSec=5.32432807682903, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4816/5000 [16:29:10<33:45, 11.01s/it][2022-12-21 02:51:40,990] [INFO] [timer.py:197:stop] 0/9750, RunningAvgSamplesPerSec=5.88109176730643, CurrSamplesPerSec=5.28770299073872, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4817/5000 [16:29:21<33:37, 11.03s/it][2022-12-21 02:51:52,022] [INFO] [timer.py:197:stop] 0/9752, RunningAvgSamplesPerSec=5.881088795477315, CurrSamplesPerSec=5.320243194169213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4818/5000 [16:29:32<33:26, 11.03s/it][2022-12-21 02:52:03,060] [INFO] [timer.py:197:stop] 0/9754, RunningAvgSamplesPerSec=5.881085351390909, CurrSamplesPerSec=5.313563470505324, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4819/5000 [16:29:43<33:15, 11.02s/it][2022-12-21 02:52:14,071] [INFO] [timer.py:197:stop] 0/9756, RunningAvgSamplesPerSec=5.8810846618543025, CurrSamplesPerSec=5.318956241590907, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4820/5000 [16:29:54<33:04, 11.02s/it][2022-12-21 02:52:25,076] [INFO] [timer.py:197:stop] 0/9758, RunningAvgSamplesPerSec=5.8810847963720425, CurrSamplesPerSec=5.339518187648851, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4821/5000 [16:30:05<32:50, 11.01s/it][2022-12-21 02:52:36,043] [INFO] [logging.py:68:log_dist] [Rank 0] step=4880, skipped=7, lr=[2.844444444444445e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:52:36,045] [INFO] [timer.py:197:stop] 0/9760, RunningAvgSamplesPerSec=5.881088717159886, CurrSamplesPerSec=5.337060776559862, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4822/5000 [16:30:16<32:40, 11.01s/it][2022-12-21 02:52:47,083] [INFO] [timer.py:197:stop] 0/9762, RunningAvgSamplesPerSec=5.8810850255846665, CurrSamplesPerSec=5.331853006007156, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4823/5000 [16:30:27<32:27, 11.00s/it][2022-12-21 02:52:58,068] [INFO] [timer.py:197:stop] 0/9764, RunningAvgSamplesPerSec=5.881087247608846, CurrSamplesPerSec=5.324186356975879, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4824/5000 [16:30:38<32:17, 11.01s/it][2022-12-21 02:53:09,090] [INFO] [timer.py:197:stop] 0/9766, RunningAvgSamplesPerSec=5.881085548329467, CurrSamplesPerSec=5.329510146511596, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 96%|█████████▋| 4825/5000 [16:30:49<32:07, 11.01s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 2.7777777777777776e-07, 'epoch': 117.67}

 96%|█████████▋| 4825/5000 [16:30:49<32:07, 11.01s/it][2022-12-21 02:53:20,111] [INFO] [timer.py:197:stop] 0/9768, RunningAvgSamplesPerSec=5.88108406596122, CurrSamplesPerSec=5.330138743612372, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4826/5000 [16:31:00<31:57, 11.02s/it][2022-12-21 02:53:31,206] [INFO] [timer.py:197:stop] 0/9770, RunningAvgSamplesPerSec=5.881074169182799, CurrSamplesPerSec=5.276026222553645, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4827/5000 [16:31:11<31:50, 11.04s/it][2022-12-21 02:53:42,232] [INFO] [timer.py:197:stop] 0/9772, RunningAvgSamplesPerSec=5.881071894573575, CurrSamplesPerSec=5.33518558806113, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4828/5000 [16:31:22<31:39, 11.04s/it][2022-12-21 02:53:53,310] [INFO] [timer.py:197:stop] 0/9774, RunningAvgSamplesPerSec=5.881063948727733, CurrSamplesPerSec=5.303637308839403, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4829/5000 [16:31:33<31:27, 11.04s/it][2022-12-21 02:54:04,341] [INFO] [timer.py:197:stop] 0/9776, RunningAvgSamplesPerSec=5.881061044887499, CurrSamplesPerSec=5.30129488551071, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4830/5000 [16:31:44<31:18, 11.05s/it][2022-12-21 02:54:15,416] [INFO] [timer.py:197:stop] 0/9778, RunningAvgSamplesPerSec=5.881054459110175, CurrSamplesPerSec=5.313531706430282, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4831/5000 [16:31:55<31:08, 11.06s/it][2022-12-21 02:54:26,445] [INFO] [logging.py:68:log_dist] [Rank 0] step=4890, skipped=7, lr=[2.6222222222222226e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:54:26,447] [INFO] [timer.py:197:stop] 0/9780, RunningAvgSamplesPerSec=5.881053646453292, CurrSamplesPerSec=5.345565600643327, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4832/5000 [16:32:07<30:55, 11.04s/it][2022-12-21 02:54:37,472] [INFO] [timer.py:197:stop] 0/9782, RunningAvgSamplesPerSec=5.881051482091528, CurrSamplesPerSec=5.330822114302117, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4833/5000 [16:32:17<30:41, 11.03s/it][2022-12-21 02:54:48,450] [INFO] [timer.py:197:stop] 0/9784, RunningAvgSamplesPerSec=5.88105439049244, CurrSamplesPerSec=5.336143065761105, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4834/5000 [16:32:29<30:30, 11.03s/it][2022-12-21 02:54:59,523] [INFO] [timer.py:197:stop] 0/9786, RunningAvgSamplesPerSec=5.881046988622946, CurrSamplesPerSec=5.29842445985513, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4835/5000 [16:32:40<30:19, 11.03s/it][2022-12-21 02:55:10,520] [INFO] [timer.py:197:stop] 0/9788, RunningAvgSamplesPerSec=5.8810479500291555, CurrSamplesPerSec=5.319647711444614, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4836/5000 [16:32:51<30:09, 11.03s/it][2022-12-21 02:55:21,553] [INFO] [timer.py:197:stop] 0/9790, RunningAvgSamplesPerSec=5.881044794251297, CurrSamplesPerSec=5.32751570308064, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4837/5000 [16:33:02<29:58, 11.03s/it][2022-12-21 02:55:32,596] [INFO] [timer.py:197:stop] 0/9792, RunningAvgSamplesPerSec=5.881040587604242, CurrSamplesPerSec=5.317163895140837, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4838/5000 [16:33:13<29:46, 11.03s/it][2022-12-21 02:55:42,685] [INFO] [timer.py:197:stop] 0/9794, RunningAvgSamplesPerSec=5.881140593095391, CurrSamplesPerSec=6.272706781419294, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 02:55:53,733] [INFO] [timer.py:197:stop] 0/9796, RunningAvgSamplesPerSec=5.881135524406425, CurrSamplesPerSec=5.302798097107958, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4839/5000 [16:33:29<33:45, 12.58s/it][2022-12-21 02:56:04,769] [INFO] [timer.py:197:stop] 0/9798, RunningAvgSamplesPerSec=5.88113211428996, CurrSamplesPerSec=5.319485157703488, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4840/5000 [16:33:40<32:18, 12.12s/it][2022-12-21 02:56:15,840] [INFO] [logging.py:68:log_dist] [Rank 0] step=4900, skipped=7, lr=[2.4000000000000003e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:56:15,842] [INFO] [timer.py:197:stop] 0/9800, RunningAvgSamplesPerSec=5.881124765324317, CurrSamplesPerSec=5.302683079792943, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4841/5000 [16:33:51<31:16, 11.80s/it][2022-12-21 02:56:26,815] [INFO] [timer.py:197:stop] 0/9802, RunningAvgSamplesPerSec=5.881128351204375, CurrSamplesPerSec=5.356661559619491, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4842/5000 [16:34:02<30:25, 11.55s/it][2022-12-21 02:56:37,834] [INFO] [timer.py:197:stop] 0/9804, RunningAvgSamplesPerSec=5.8811268308221205, CurrSamplesPerSec=5.307711414268036, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4843/5000 [16:34:13<29:48, 11.39s/it][2022-12-21 02:56:48,894] [INFO] [timer.py:197:stop] 0/9806, RunningAvgSamplesPerSec=5.8811208157542385, CurrSamplesPerSec=5.291951897858474, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4844/5000 [16:34:24<29:21, 11.29s/it][2022-12-21 02:56:59,940] [INFO] [timer.py:197:stop] 0/9808, RunningAvgSamplesPerSec=5.881116364077755, CurrSamplesPerSec=5.271964977722606, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4845/5000 [16:34:35<28:59, 11.22s/it][2022-12-21 02:57:10,989] [INFO] [timer.py:197:stop] 0/9810, RunningAvgSamplesPerSec=5.8811116763637346, CurrSamplesPerSec=5.318735135662933, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4846/5000 [16:34:46<28:39, 11.17s/it][2022-12-21 02:57:21,928] [INFO] [timer.py:197:stop] 0/9812, RunningAvgSamplesPerSec=5.881118962992291, CurrSamplesPerSec=5.364329681248229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4847/5000 [16:34:57<28:18, 11.10s/it][2022-12-21 02:57:32,912] [INFO] [timer.py:197:stop] 0/9814, RunningAvgSamplesPerSec=5.881121381032901, CurrSamplesPerSec=5.358652001922791, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4848/5000 [16:35:08<28:01, 11.06s/it][2022-12-21 02:57:43,980] [INFO] [timer.py:197:stop] 0/9816, RunningAvgSamplesPerSec=5.881114649910023, CurrSamplesPerSec=5.300976842569895, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4849/5000 [16:35:19<27:50, 11.07s/it][2022-12-21 02:57:54,963] [INFO] [timer.py:197:stop] 0/9818, RunningAvgSamplesPerSec=5.881117101706607, CurrSamplesPerSec=5.325304480290763, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4850/5000 [16:35:30<27:36, 11.04s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 2.2e-07, 'epoch': 118.29}

 97%|█████████▋| 4850/5000 [16:35:30<27:36, 11.04s/it][2022-12-21 02:58:05,988] [INFO] [logging.py:68:log_dist] [Rank 0] step=4910, skipped=7, lr=[2.177777777777778e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:58:05,990] [INFO] [timer.py:197:stop] 0/9820, RunningAvgSamplesPerSec=5.8811148483372495, CurrSamplesPerSec=5.33874105804406, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4851/5000 [16:35:41<27:24, 11.04s/it][2022-12-21 02:58:17,022] [INFO] [timer.py:197:stop] 0/9822, RunningAvgSamplesPerSec=5.8811119471842925, CurrSamplesPerSec=5.316668084125516, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4852/5000 [16:35:52<27:13, 11.04s/it][2022-12-21 02:58:28,067] [INFO] [timer.py:197:stop] 0/9824, RunningAvgSamplesPerSec=5.881107637965968, CurrSamplesPerSec=5.317996914146575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4853/5000 [16:36:03<27:02, 11.04s/it][2022-12-21 02:58:39,105] [INFO] [timer.py:197:stop] 0/9826, RunningAvgSamplesPerSec=5.881104086336548, CurrSamplesPerSec=5.303822159698251, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4854/5000 [16:36:14<26:51, 11.04s/it][2022-12-21 02:58:50,113] [INFO] [timer.py:197:stop] 0/9828, RunningAvgSamplesPerSec=5.881103804143445, CurrSamplesPerSec=5.344964010484144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4855/5000 [16:36:25<26:39, 11.03s/it][2022-12-21 02:59:01,121] [INFO] [timer.py:197:stop] 0/9830, RunningAvgSamplesPerSec=5.8811033900707255, CurrSamplesPerSec=5.336411662423746, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4856/5000 [16:36:36<26:27, 11.02s/it][2022-12-21 02:59:12,185] [INFO] [timer.py:197:stop] 0/9832, RunningAvgSamplesPerSec=5.8810968217213055, CurrSamplesPerSec=5.309499493073067, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4857/5000 [16:36:47<26:18, 11.04s/it][2022-12-21 02:59:23,190] [INFO] [timer.py:197:stop] 0/9834, RunningAvgSamplesPerSec=5.881096721686233, CurrSamplesPerSec=5.33305551068884, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4858/5000 [16:36:58<26:05, 11.03s/it][2022-12-21 02:59:34,316] [INFO] [timer.py:197:stop] 0/9836, RunningAvgSamplesPerSec=5.881083337539888, CurrSamplesPerSec=5.248332115010413, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4859/5000 [16:37:09<25:58, 11.06s/it][2022-12-21 02:59:45,315] [INFO] [timer.py:197:stop] 0/9838, RunningAvgSamplesPerSec=5.881083996184068, CurrSamplesPerSec=5.336562521222161, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4860/5000 [16:37:20<25:45, 11.04s/it][2022-12-21 02:59:56,283] [INFO] [logging.py:68:log_dist] [Rank 0] step=4920, skipped=7, lr=[1.9555555555555558e-07], mom=[[0.9, 0.999]]
[2022-12-21 02:59:56,284] [INFO] [timer.py:197:stop] 0/9840, RunningAvgSamplesPerSec=5.881088101585368, CurrSamplesPerSec=5.344239982814685, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4861/5000 [16:37:31<25:31, 11.02s/it][2022-12-21 03:00:07,294] [INFO] [timer.py:197:stop] 0/9842, RunningAvgSamplesPerSec=5.88108773628177, CurrSamplesPerSec=5.3468501273095335, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4862/5000 [16:37:42<25:20, 11.02s/it][2022-12-21 03:00:18,285] [INFO] [timer.py:197:stop] 0/9844, RunningAvgSamplesPerSec=5.881089347327349, CurrSamplesPerSec=5.300731270017787, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4863/5000 [16:37:53<25:08, 11.01s/it][2022-12-21 03:00:29,279] [INFO] [timer.py:197:stop] 0/9846, RunningAvgSamplesPerSec=5.881090795756704, CurrSamplesPerSec=5.361119062076674, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4864/5000 [16:38:04<24:56, 11.00s/it][2022-12-21 03:00:40,260] [INFO] [timer.py:197:stop] 0/9848, RunningAvgSamplesPerSec=5.881093604153213, CurrSamplesPerSec=5.319455431051029, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4865/5000 [16:38:15<24:44, 11.00s/it][2022-12-21 03:00:51,279] [INFO] [timer.py:197:stop] 0/9850, RunningAvgSamplesPerSec=5.881092144237029, CurrSamplesPerSec=5.327926823802811, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4866/5000 [16:38:26<24:34, 11.00s/it][2022-12-21 03:01:02,244] [INFO] [timer.py:197:stop] 0/9852, RunningAvgSamplesPerSec=5.881096722359585, CurrSamplesPerSec=5.333900935916359, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4867/5000 [16:38:37<24:21, 10.99s/it][2022-12-21 03:01:13,203] [INFO] [timer.py:197:stop] 0/9854, RunningAvgSamplesPerSec=5.8811017963926435, CurrSamplesPerSec=5.379798822099924, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4868/5000 [16:38:48<24:09, 10.98s/it][2022-12-21 03:01:24,132] [INFO] [timer.py:197:stop] 0/9856, RunningAvgSamplesPerSec=5.88111031772632, CurrSamplesPerSec=5.344157632304582, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4869/5000 [16:38:59<23:56, 10.97s/it][2022-12-21 03:01:35,117] [INFO] [timer.py:197:stop] 0/9858, RunningAvgSamplesPerSec=5.881112518386495, CurrSamplesPerSec=5.344831193539486, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4870/5000 [16:39:10<23:46, 10.97s/it][2022-12-21 03:01:46,108] [INFO] [logging.py:68:log_dist] [Rank 0] step=4930, skipped=7, lr=[1.7333333333333335e-07], mom=[[0.9, 0.999]]
[2022-12-21 03:01:46,110] [INFO] [timer.py:197:stop] 0/9860, RunningAvgSamplesPerSec=5.881113800008539, CurrSamplesPerSec=5.319881333959638, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4871/5000 [16:39:21<23:36, 10.98s/it][2022-12-21 03:01:57,108] [INFO] [timer.py:197:stop] 0/9862, RunningAvgSamplesPerSec=5.88111467715933, CurrSamplesPerSec=5.350249475250512, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4872/5000 [16:39:32<23:25, 10.98s/it][2022-12-21 03:02:08,071] [INFO] [timer.py:197:stop] 0/9864, RunningAvgSamplesPerSec=5.8811193704716205, CurrSamplesPerSec=5.333700841158969, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4873/5000 [16:39:43<23:14, 10.98s/it][2022-12-21 03:02:19,064] [INFO] [timer.py:197:stop] 0/9866, RunningAvgSamplesPerSec=5.881120730001057, CurrSamplesPerSec=5.346225248942059, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 97%|█████████▋| 4874/5000 [16:39:54<23:03, 10.98s/it][2022-12-21 03:02:29,995] [INFO] [timer.py:197:stop] 0/9868, RunningAvgSamplesPerSec=5.881128856960425, CurrSamplesPerSec=5.335659193251529, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4875/5000 [16:40:05<22:50, 10.97s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 1.6444444444444447e-07, 'epoch': 118.89}

 98%|█████████▊| 4875/5000 [16:40:05<22:50, 10.97s/it][2022-12-21 03:02:40,974] [INFO] [timer.py:197:stop] 0/9870, RunningAvgSamplesPerSec=5.881131866184139, CurrSamplesPerSec=5.365476313211763, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4876/5000 [16:40:16<22:40, 10.97s/it][2022-12-21 03:02:51,918] [INFO] [timer.py:197:stop] 0/9872, RunningAvgSamplesPerSec=5.881138587510723, CurrSamplesPerSec=5.3696369932724375, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4877/5000 [16:40:27<22:28, 10.96s/it][2022-12-21 03:03:02,872] [INFO] [timer.py:197:stop] 0/9874, RunningAvgSamplesPerSec=5.881144273738245, CurrSamplesPerSec=5.356830028003102, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4878/5000 [16:40:38<22:17, 10.96s/it][2022-12-21 03:03:13,842] [INFO] [timer.py:197:stop] 0/9876, RunningAvgSamplesPerSec=5.881148240551594, CurrSamplesPerSec=5.328577894729863, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4879/5000 [16:40:49<22:06, 10.96s/it][2022-12-21 03:03:23,928] [INFO] [timer.py:197:stop] 0/9878, RunningAvgSamplesPerSec=5.881247832402207, CurrSamplesPerSec=5.3351493235073555, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4880/5000 [16:41:04<24:20, 12.17s/it][2022-12-21 03:03:34,871] [INFO] [logging.py:68:log_dist] [Rank 0] step=4940, skipped=7, lr=[1.5111111111111112e-07], mom=[[0.9, 0.999]]
[2022-12-21 03:03:34,872] [INFO] [timer.py:197:stop] 0/9880, RunningAvgSamplesPerSec=5.881254335323087, CurrSamplesPerSec=5.347510518801368, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4881/5000 [16:41:15<23:24, 11.80s/it][2022-12-21 03:03:45,841] [INFO] [timer.py:197:stop] 0/9882, RunningAvgSamplesPerSec=5.881258195066024, CurrSamplesPerSec=5.330498189994769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4882/5000 [16:41:26<22:45, 11.57s/it][2022-12-21 03:03:56,838] [INFO] [timer.py:197:stop] 0/9884, RunningAvgSamplesPerSec=5.881259028773293, CurrSamplesPerSec=5.361381398359978, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4883/5000 [16:41:37<22:11, 11.38s/it][2022-12-21 03:04:07,823] [INFO] [timer.py:197:stop] 0/9886, RunningAvgSamplesPerSec=5.881261110617577, CurrSamplesPerSec=5.3142110355566015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4884/5000 [16:41:48<21:47, 11.27s/it][2022-12-21 03:04:18,798] [INFO] [timer.py:197:stop] 0/9888, RunningAvgSamplesPerSec=5.881264327620437, CurrSamplesPerSec=5.341422782202486, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4885/5000 [16:41:59<21:26, 11.19s/it][2022-12-21 03:04:29,759] [INFO] [timer.py:197:stop] 0/9890, RunningAvgSamplesPerSec=5.881269080466366, CurrSamplesPerSec=5.384018468835765, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4886/5000 [16:42:10<21:07, 11.12s/it][2022-12-21 03:04:40,756] [INFO] [timer.py:197:stop] 0/9892, RunningAvgSamplesPerSec=5.8812699399123884, CurrSamplesPerSec=5.356439445548449, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4887/5000 [16:42:21<20:50, 11.07s/it][2022-12-21 03:04:51,737] [INFO] [timer.py:197:stop] 0/9894, RunningAvgSamplesPerSec=5.881272418274777, CurrSamplesPerSec=5.321790300863083, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4888/5000 [16:42:32<20:38, 11.05s/it][2022-12-21 03:05:02,759] [INFO] [timer.py:197:stop] 0/9896, RunningAvgSamplesPerSec=5.881270562505185, CurrSamplesPerSec=5.3222288183940485, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4889/5000 [16:42:43<20:26, 11.05s/it][2022-12-21 03:05:13,798] [INFO] [timer.py:197:stop] 0/9898, RunningAvgSamplesPerSec=5.881266913513807, CurrSamplesPerSec=5.326018102339611, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4890/5000 [16:42:54<20:14, 11.04s/it][2022-12-21 03:05:24,774] [INFO] [logging.py:68:log_dist] [Rank 0] step=4950, skipped=7, lr=[1.288888888888889e-07], mom=[[0.9, 0.999]]
[2022-12-21 03:05:24,775] [INFO] [timer.py:197:stop] 0/9900, RunningAvgSamplesPerSec=5.881270029192826, CurrSamplesPerSec=5.361373260195539, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4891/5000 [16:43:05<20:01, 11.02s/it][2022-12-21 03:05:35,763] [INFO] [timer.py:197:stop] 0/9902, RunningAvgSamplesPerSec=5.881272010755754, CurrSamplesPerSec=5.353279469650216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4892/5000 [16:43:16<19:48, 11.01s/it][2022-12-21 03:05:46,730] [INFO] [timer.py:197:stop] 0/9904, RunningAvgSamplesPerSec=5.881276049252223, CurrSamplesPerSec=5.352018742511104, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4893/5000 [16:43:27<19:36, 11.00s/it][2022-12-21 03:05:57,774] [INFO] [timer.py:197:stop] 0/9906, RunningAvgSamplesPerSec=5.881271752456181, CurrSamplesPerSec=5.288471999742152, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4894/5000 [16:43:38<19:26, 11.01s/it][2022-12-21 03:06:08,789] [INFO] [timer.py:197:stop] 0/9908, RunningAvgSamplesPerSec=5.881270703082028, CurrSamplesPerSec=5.310281998867976, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4895/5000 [16:43:49<19:17, 11.03s/it][2022-12-21 03:06:19,834] [INFO] [timer.py:197:stop] 0/9910, RunningAvgSamplesPerSec=5.8812664413084175, CurrSamplesPerSec=5.328548701008293, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4896/5000 [16:44:00<19:05, 11.02s/it][2022-12-21 03:06:30,841] [INFO] [timer.py:197:stop] 0/9912, RunningAvgSamplesPerSec=5.881266277604089, CurrSamplesPerSec=5.316582579765042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4897/5000 [16:44:11<18:54, 11.02s/it][2022-12-21 03:06:41,851] [INFO] [timer.py:197:stop] 0/9914, RunningAvgSamplesPerSec=5.881265718482466, CurrSamplesPerSec=5.322109580054575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4898/5000 [16:44:22<18:44, 11.03s/it][2022-12-21 03:06:52,895] [INFO] [timer.py:197:stop] 0/9916, RunningAvgSamplesPerSec=5.881261520192311, CurrSamplesPerSec=5.3238000977990145, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4899/5000 [16:44:33<18:34, 11.03s/it][2022-12-21 03:07:03,937] [INFO] [timer.py:197:stop] 0/9918, RunningAvgSamplesPerSec=5.881257429248042, CurrSamplesPerSec=5.32185191686462, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4900/5000 [16:44:44<18:22, 11.03s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 1.088888888888889e-07, 'epoch': 119.51}

 98%|█████████▊| 4900/5000 [16:44:44<18:22, 11.03s/it][2022-12-21 03:07:14,992] [INFO] [logging.py:68:log_dist] [Rank 0] step=4960, skipped=7, lr=[1.0666666666666667e-07], mom=[[0.9, 0.999]]
[2022-12-21 03:07:14,993] [INFO] [timer.py:197:stop] 0/9920, RunningAvgSamplesPerSec=5.881252001212702, CurrSamplesPerSec=5.296224357890981, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4901/5000 [16:44:55<18:12, 11.04s/it][2022-12-21 03:07:26,028] [INFO] [timer.py:197:stop] 0/9922, RunningAvgSamplesPerSec=5.881248696874494, CurrSamplesPerSec=5.324292382146269, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4902/5000 [16:45:06<18:01, 11.04s/it][2022-12-21 03:07:37,064] [INFO] [timer.py:197:stop] 0/9924, RunningAvgSamplesPerSec=5.881245278861728, CurrSamplesPerSec=5.314292465724569, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4903/5000 [16:45:17<17:49, 11.02s/it][2022-12-21 03:07:48,010] [INFO] [timer.py:197:stop] 0/9926, RunningAvgSamplesPerSec=5.881251677516267, CurrSamplesPerSec=5.3551759477589, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4904/5000 [16:45:28<17:37, 11.01s/it][2022-12-21 03:07:59,039] [INFO] [timer.py:197:stop] 0/9928, RunningAvgSamplesPerSec=5.881249024705565, CurrSamplesPerSec=5.317215714203256, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4905/5000 [16:45:39<17:26, 11.02s/it][2022-12-21 03:08:10,081] [INFO] [timer.py:197:stop] 0/9930, RunningAvgSamplesPerSec=5.88124487811138, CurrSamplesPerSec=5.310954823118706, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4906/5000 [16:45:50<17:16, 11.03s/it][2022-12-21 03:08:21,123] [INFO] [timer.py:197:stop] 0/9932, RunningAvgSamplesPerSec=5.881240674099582, CurrSamplesPerSec=5.320687573546213, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4907/5000 [16:46:01<17:05, 11.03s/it][2022-12-21 03:08:32,129] [INFO] [timer.py:197:stop] 0/9934, RunningAvgSamplesPerSec=5.881240539041054, CurrSamplesPerSec=5.343051998027551, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4908/5000 [16:46:12<16:54, 11.02s/it][2022-12-21 03:08:43,162] [INFO] [timer.py:197:stop] 0/9936, RunningAvgSamplesPerSec=5.881237427243262, CurrSamplesPerSec=5.324150875391776, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4909/5000 [16:46:23<16:42, 11.02s/it][2022-12-21 03:08:54,223] [INFO] [timer.py:197:stop] 0/9938, RunningAvgSamplesPerSec=5.88123135254784, CurrSamplesPerSec=5.279598992288359, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4910/5000 [16:46:34<16:32, 11.03s/it][2022-12-21 03:09:05,271] [INFO] [logging.py:68:log_dist] [Rank 0] step=4970, skipped=7, lr=[8.444444444444444e-08], mom=[[0.9, 0.999]]
[2022-12-21 03:09:05,273] [INFO] [timer.py:197:stop] 0/9940, RunningAvgSamplesPerSec=5.881226305924845, CurrSamplesPerSec=5.28592498026212, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4911/5000 [16:46:45<16:23, 11.05s/it][2022-12-21 03:09:16,333] [INFO] [timer.py:197:stop] 0/9942, RunningAvgSamplesPerSec=5.881220200700927, CurrSamplesPerSec=5.319063112532863, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4912/5000 [16:46:56<16:10, 11.03s/it][2022-12-21 03:09:27,297] [INFO] [timer.py:197:stop] 0/9944, RunningAvgSamplesPerSec=5.881224499405726, CurrSamplesPerSec=5.344868228369991, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4913/5000 [16:47:07<15:58, 11.02s/it][2022-12-21 03:09:38,345] [INFO] [timer.py:197:stop] 0/9946, RunningAvgSamplesPerSec=5.881219741902373, CurrSamplesPerSec=5.2998856528932325, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4914/5000 [16:47:18<15:48, 11.03s/it][2022-12-21 03:09:49,392] [INFO] [timer.py:197:stop] 0/9948, RunningAvgSamplesPerSec=5.881215129864212, CurrSamplesPerSec=5.308986840514734, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4915/5000 [16:47:29<15:37, 11.03s/it][2022-12-21 03:10:00,339] [INFO] [timer.py:197:stop] 0/9950, RunningAvgSamplesPerSec=5.88122133864681, CurrSamplesPerSec=5.373435378821797, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4916/5000 [16:47:40<15:24, 11.01s/it][2022-12-21 03:10:11,321] [INFO] [timer.py:197:stop] 0/9952, RunningAvgSamplesPerSec=5.88122377502043, CurrSamplesPerSec=5.362875598439042, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4917/5000 [16:47:51<15:12, 11.00s/it][2022-12-21 03:10:22,346] [INFO] [timer.py:197:stop] 0/9954, RunningAvgSamplesPerSec=5.88122158669117, CurrSamplesPerSec=5.306522408417547, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4918/5000 [16:48:02<15:02, 11.01s/it][2022-12-21 03:10:33,386] [INFO] [timer.py:197:stop] 0/9956, RunningAvgSamplesPerSec=5.881217656529153, CurrSamplesPerSec=5.311343003845229, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4919/5000 [16:48:13<14:52, 11.02s/it][2022-12-21 03:10:44,404] [INFO] [timer.py:197:stop] 0/9958, RunningAvgSamplesPerSec=5.8812160776662035, CurrSamplesPerSec=5.3289809277062385, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4920/5000 [16:48:24<14:41, 11.02s/it][2022-12-21 03:10:54,527] [INFO] [logging.py:68:log_dist] [Rank 0] step=4980, skipped=7, lr=[6.222222222222223e-08], mom=[[0.9, 0.999]]
[2022-12-21 03:10:54,528] [INFO] [timer.py:197:stop] 0/9960, RunningAvgSamplesPerSec=5.881310406137595, CurrSamplesPerSec=6.264844214324032, MemAllocated=3.0GB, MaxMemAllocated=19.53GB
[2022-12-21 03:11:05,525] [INFO] [timer.py:197:stop] 0/9962, RunningAvgSamplesPerSec=5.881310809623745, CurrSamplesPerSec=5.313677908467399, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4921/5000 [16:48:41<16:32, 12.56s/it][2022-12-21 03:11:16,549] [INFO] [timer.py:197:stop] 0/9964, RunningAvgSamplesPerSec=5.881308695669039, CurrSamplesPerSec=5.334383005485661, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4922/5000 [16:48:52<15:43, 12.10s/it][2022-12-21 03:11:27,575] [INFO] [timer.py:197:stop] 0/9966, RunningAvgSamplesPerSec=5.881306452931631, CurrSamplesPerSec=5.307873249226387, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4923/5000 [16:49:03<15:06, 11.78s/it][2022-12-21 03:11:38,617] [INFO] [timer.py:197:stop] 0/9968, RunningAvgSamplesPerSec=5.881302473662925, CurrSamplesPerSec=5.298105506400605, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4924/5000 [16:49:14<14:38, 11.56s/it][2022-12-21 03:11:49,641] [INFO] [timer.py:197:stop] 0/9970, RunningAvgSamplesPerSec=5.8813003621421105, CurrSamplesPerSec=5.307426390440144, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 98%|█████████▊| 4925/5000 [16:49:25<14:14, 11.40s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 5.111111111111112e-08, 'epoch': 120.12}

 98%|█████████▊| 4925/5000 [16:49:25<14:14, 11.40s/it][2022-12-21 03:12:00,672] [INFO] [timer.py:197:stop] 0/9972, RunningAvgSamplesPerSec=5.881297828453425, CurrSamplesPerSec=5.326023174664336, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4926/5000 [16:49:36<13:55, 11.29s/it][2022-12-21 03:12:11,679] [INFO] [timer.py:197:stop] 0/9974, RunningAvgSamplesPerSec=5.881297694126449, CurrSamplesPerSec=5.321831237362568, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4927/5000 [16:49:47<13:37, 11.20s/it][2022-12-21 03:12:22,736] [INFO] [timer.py:197:stop] 0/9976, RunningAvgSamplesPerSec=5.881292137187019, CurrSamplesPerSec=5.320043279296575, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4928/5000 [16:49:58<13:23, 11.16s/it][2022-12-21 03:12:33,790] [INFO] [timer.py:197:stop] 0/9978, RunningAvgSamplesPerSec=5.881286867633184, CurrSamplesPerSec=5.318340605262924, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4929/5000 [16:50:09<13:10, 11.13s/it][2022-12-21 03:12:44,850] [INFO] [logging.py:68:log_dist] [Rank 0] step=4990, skipped=7, lr=[4e-08], mom=[[0.9, 0.999]]
[2022-12-21 03:12:44,851] [INFO] [timer.py:197:stop] 0/9980, RunningAvgSamplesPerSec=5.881280658512985, CurrSamplesPerSec=5.317547506526408, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4930/5000 [16:50:20<12:57, 11.11s/it][2022-12-21 03:12:55,857] [INFO] [timer.py:197:stop] 0/9982, RunningAvgSamplesPerSec=5.881280511208736, CurrSamplesPerSec=5.339307051075701, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4931/5000 [16:50:31<12:44, 11.08s/it][2022-12-21 03:13:06,894] [INFO] [timer.py:197:stop] 0/9984, RunningAvgSamplesPerSec=5.881276949274559, CurrSamplesPerSec=5.319455009398775, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4932/5000 [16:50:42<12:32, 11.07s/it][2022-12-21 03:13:17,951] [INFO] [timer.py:197:stop] 0/9986, RunningAvgSamplesPerSec=5.881271301530712, CurrSamplesPerSec=5.315437170821566, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4933/5000 [16:50:53<12:21, 11.06s/it][2022-12-21 03:13:28,989] [INFO] [timer.py:197:stop] 0/9988, RunningAvgSamplesPerSec=5.881267827397466, CurrSamplesPerSec=5.316848368341264, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4934/5000 [16:51:04<12:09, 11.06s/it][2022-12-21 03:13:39,953] [INFO] [timer.py:197:stop] 0/9990, RunningAvgSamplesPerSec=5.88127229282922, CurrSamplesPerSec=5.340523120694523, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4935/5000 [16:51:15<11:56, 11.03s/it][2022-12-21 03:13:50,984] [INFO] [timer.py:197:stop] 0/9992, RunningAvgSamplesPerSec=5.8812693415513495, CurrSamplesPerSec=5.325216162522858, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4936/5000 [16:51:26<11:45, 11.03s/it][2022-12-21 03:14:02,014] [INFO] [timer.py:197:stop] 0/9994, RunningAvgSamplesPerSec=5.8812666873656925, CurrSamplesPerSec=5.30069149484874, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▊| 4937/5000 [16:51:37<11:34, 11.03s/it][2022-12-21 03:14:13,010] [INFO] [timer.py:197:stop] 0/9996, RunningAvgSamplesPerSec=5.8812675051848675, CurrSamplesPerSec=5.309296184350492, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4938/5000 [16:51:48<11:23, 11.02s/it][2022-12-21 03:14:23,956] [INFO] [timer.py:197:stop] 0/9998, RunningAvgSamplesPerSec=5.881273955265471, CurrSamplesPerSec=5.350657499779842, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4939/5000 [16:51:59<11:10, 11.00s/it][2022-12-21 03:14:34,929] [INFO] [logging.py:68:log_dist] [Rank 0] step=5000, skipped=7, lr=[1.777777777777778e-08], mom=[[0.9, 0.999]]
[2022-12-21 03:14:34,931] [INFO] [timer.py:197:stop] 0/10000, RunningAvgSamplesPerSec=5.8812771123839065, CurrSamplesPerSec=5.330708418737848, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4940/5000 [16:52:10<10:59, 10.99s/it][2022-12-21 03:14:45,963] [INFO] [timer.py:197:stop] 0/10002, RunningAvgSamplesPerSec=5.88127412422575, CurrSamplesPerSec=5.329034458456789, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4941/5000 [16:52:21<10:49, 11.00s/it][2022-12-21 03:14:56,917] [INFO] [timer.py:197:stop] 0/10004, RunningAvgSamplesPerSec=5.8812795926481956, CurrSamplesPerSec=5.323810867524946, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4942/5000 [16:52:32<10:37, 10.99s/it][2022-12-21 03:15:07,892] [INFO] [timer.py:197:stop] 0/10006, RunningAvgSamplesPerSec=5.881282877952258, CurrSamplesPerSec=5.3429847854519625, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4943/5000 [16:52:43<10:26, 10.98s/it][2022-12-21 03:15:18,928] [INFO] [timer.py:197:stop] 0/10008, RunningAvgSamplesPerSec=5.881279497163925, CurrSamplesPerSec=5.344004854359268, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4944/5000 [16:52:54<10:15, 11.00s/it][2022-12-21 03:15:29,923] [INFO] [timer.py:197:stop] 0/10010, RunningAvgSamplesPerSec=5.881280660743498, CurrSamplesPerSec=5.324802291250986, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4945/5000 [16:53:05<10:04, 11.00s/it][2022-12-21 03:15:40,924] [INFO] [timer.py:197:stop] 0/10012, RunningAvgSamplesPerSec=5.881281027707231, CurrSamplesPerSec=5.353477192552662, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4946/5000 [16:53:16<09:53, 11.00s/it][2022-12-21 03:15:51,923] [INFO] [timer.py:197:stop] 0/10014, RunningAvgSamplesPerSec=5.881281732983303, CurrSamplesPerSec=5.3478328916973386, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4947/5000 [16:53:27<09:42, 11.00s/it][2022-12-21 03:16:02,904] [INFO] [timer.py:197:stop] 0/10016, RunningAvgSamplesPerSec=5.881284430933788, CurrSamplesPerSec=5.348701766173452, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4948/5000 [16:53:38<09:31, 10.99s/it][2022-12-21 03:16:13,889] [INFO] [timer.py:197:stop] 0/10018, RunningAvgSamplesPerSec=5.881286523101808, CurrSamplesPerSec=5.326754535924154, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4949/5000 [16:53:49<09:20, 10.99s/it][2022-12-21 03:16:24,837] [INFO] [logging.py:68:log_dist] [Rank 0] step=5010, skipped=7, lr=[0.0], mom=[[0.9, 0.999]]
[2022-12-21 03:16:24,838] [INFO] [timer.py:197:stop] 0/10020, RunningAvgSamplesPerSec=5.881292564011672, CurrSamplesPerSec=5.346546614971619, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4950/5000 [16:54:00<09:08, 10.98s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 0.0, 'epoch': 120.72}

 99%|█████████▉| 4950/5000 [16:54:00<09:08, 10.98s/it][2022-12-21 03:16:35,760] [INFO] [timer.py:197:stop] 0/10022, RunningAvgSamplesPerSec=5.881301597989175, CurrSamplesPerSec=5.372602537606953, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4951/5000 [16:54:11<08:57, 10.96s/it][2022-12-21 03:16:46,806] [INFO] [timer.py:197:stop] 0/10024, RunningAvgSamplesPerSec=5.881297158673873, CurrSamplesPerSec=5.307300049463672, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4952/5000 [16:54:22<08:47, 10.99s/it][2022-12-21 03:16:57,805] [INFO] [timer.py:197:stop] 0/10026, RunningAvgSamplesPerSec=5.881297853823559, CurrSamplesPerSec=5.322416444935216, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4953/5000 [16:54:33<08:36, 10.99s/it][2022-12-21 03:17:08,782] [INFO] [timer.py:197:stop] 0/10028, RunningAvgSamplesPerSec=5.8813008304838865, CurrSamplesPerSec=5.338586466179769, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4954/5000 [16:54:44<08:25, 10.99s/it][2022-12-21 03:17:19,821] [INFO] [timer.py:197:stop] 0/10030, RunningAvgSamplesPerSec=5.88129711425438, CurrSamplesPerSec=5.302242749962984, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4955/5000 [16:54:55<08:15, 11.00s/it][2022-12-21 03:17:30,871] [INFO] [timer.py:197:stop] 0/10032, RunningAvgSamplesPerSec=5.881292074051597, CurrSamplesPerSec=5.2975229141873985, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4956/5000 [16:55:06<08:04, 11.02s/it][2022-12-21 03:17:41,877] [INFO] [timer.py:197:stop] 0/10034, RunningAvgSamplesPerSec=5.8812919482005, CurrSamplesPerSec=5.335591742326389, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4957/5000 [16:55:17<07:53, 11.01s/it][2022-12-21 03:17:52,832] [INFO] [timer.py:197:stop] 0/10036, RunningAvgSamplesPerSec=5.88129724258707, CurrSamplesPerSec=5.34237399424286, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4958/5000 [16:55:28<07:41, 11.00s/it][2022-12-21 03:18:03,877] [INFO] [timer.py:197:stop] 0/10038, RunningAvgSamplesPerSec=5.881292815474794, CurrSamplesPerSec=5.311328291034376, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4959/5000 [16:55:39<07:31, 11.01s/it][2022-12-21 03:18:14,888] [INFO] [logging.py:68:log_dist] [Rank 0] step=5020, skipped=7, lr=[0.0], mom=[[0.9, 0.999]]
[2022-12-21 03:18:14,889] [INFO] [timer.py:197:stop] 0/10040, RunningAvgSamplesPerSec=5.881291976412807, CurrSamplesPerSec=5.338066270856126, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4960/5000 [16:55:50<07:20, 11.01s/it][2022-12-21 03:18:25,892] [INFO] [timer.py:197:stop] 0/10042, RunningAvgSamplesPerSec=5.8812922555520055, CurrSamplesPerSec=5.331409300992028, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4961/5000 [16:56:01<07:09, 11.01s/it][2022-12-21 03:18:35,941] [INFO] [timer.py:197:stop] 0/10044, RunningAvgSamplesPerSec=5.881394103001211, CurrSamplesPerSec=5.365999933953345, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4962/5000 [16:56:16<07:43, 12.21s/it][2022-12-21 03:18:46,935] [INFO] [timer.py:197:stop] 0/10046, RunningAvgSamplesPerSec=5.881395138690692, CurrSamplesPerSec=5.351609016676832, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4963/5000 [16:56:27<07:18, 11.85s/it][2022-12-21 03:18:57,982] [INFO] [timer.py:197:stop] 0/10048, RunningAvgSamplesPerSec=5.881390553685634, CurrSamplesPerSec=5.315996759333798, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4964/5000 [16:56:38<06:57, 11.60s/it][2022-12-21 03:19:08,986] [INFO] [timer.py:197:stop] 0/10050, RunningAvgSamplesPerSec=5.88139055519217, CurrSamplesPerSec=5.330042857097472, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4965/5000 [16:56:49<06:39, 11.41s/it][2022-12-21 03:19:19,931] [INFO] [timer.py:197:stop] 0/10052, RunningAvgSamplesPerSec=5.881396854770055, CurrSamplesPerSec=5.350455505995745, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4966/5000 [16:57:00<06:23, 11.28s/it][2022-12-21 03:19:30,932] [INFO] [timer.py:197:stop] 0/10054, RunningAvgSamplesPerSec=5.881397241758793, CurrSamplesPerSec=5.332520926585494, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4967/5000 [16:57:11<06:09, 11.19s/it][2022-12-21 03:19:41,928] [INFO] [timer.py:197:stop] 0/10056, RunningAvgSamplesPerSec=5.881398091923873, CurrSamplesPerSec=5.325531204145525, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4968/5000 [16:57:22<05:56, 11.14s/it][2022-12-21 03:19:52,969] [INFO] [timer.py:197:stop] 0/10058, RunningAvgSamplesPerSec=5.88139417474471, CurrSamplesPerSec=5.3184107818715285, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4969/5000 [16:57:33<05:44, 11.11s/it][2022-12-21 03:20:04,002] [INFO] [logging.py:68:log_dist] [Rank 0] step=5030, skipped=7, lr=[0.0], mom=[[0.9, 0.999]]
[2022-12-21 03:20:04,004] [INFO] [timer.py:197:stop] 0/10060, RunningAvgSamplesPerSec=5.881390781540597, CurrSamplesPerSec=5.316073822511502, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4970/5000 [16:57:44<05:32, 11.09s/it][2022-12-21 03:20:15,014] [INFO] [timer.py:197:stop] 0/10062, RunningAvgSamplesPerSec=5.88139012696381, CurrSamplesPerSec=5.337560398528682, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4971/5000 [16:57:55<05:20, 11.07s/it][2022-12-21 03:20:26,047] [INFO] [timer.py:197:stop] 0/10064, RunningAvgSamplesPerSec=5.881387001527627, CurrSamplesPerSec=5.326416943356395, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4972/5000 [16:58:06<05:09, 11.05s/it][2022-12-21 03:20:37,124] [INFO] [timer.py:197:stop] 0/10066, RunningAvgSamplesPerSec=5.881379125770512, CurrSamplesPerSec=5.271700551656549, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4973/5000 [16:58:17<04:58, 11.07s/it][2022-12-21 03:20:48,159] [INFO] [timer.py:197:stop] 0/10068, RunningAvgSamplesPerSec=5.881375895165819, CurrSamplesPerSec=5.328140022491944, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

 99%|█████████▉| 4974/5000 [16:58:28<04:47, 11.05s/it][2022-12-21 03:20:59,157] [INFO] [timer.py:197:stop] 0/10070, RunningAvgSamplesPerSec=5.881376712110741, CurrSamplesPerSec=5.354996900736099, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4975/5000 [16:58:39<04:35, 11.03s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 0.0, 'epoch': 121.34}

100%|█████████▉| 4975/5000 [16:58:39<04:35, 11.03s/it][2022-12-21 03:21:10,127] [INFO] [timer.py:197:stop] 0/10072, RunningAvgSamplesPerSec=5.881380590559962, CurrSamplesPerSec=5.353564101299987, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4976/5000 [16:58:50<04:24, 11.02s/it][2022-12-21 03:21:21,154] [INFO] [timer.py:197:stop] 0/10074, RunningAvgSamplesPerSec=5.881378389499504, CurrSamplesPerSec=5.3343020184924015, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4977/5000 [16:59:01<04:13, 11.02s/it][2022-12-21 03:21:32,189] [INFO] [timer.py:197:stop] 0/10076, RunningAvgSamplesPerSec=5.8813751060673045, CurrSamplesPerSec=5.327441056792944, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4978/5000 [16:59:12<04:02, 11.02s/it][2022-12-21 03:21:43,167] [INFO] [timer.py:197:stop] 0/10078, RunningAvgSamplesPerSec=5.881378003258725, CurrSamplesPerSec=5.348466458088087, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4979/5000 [16:59:23<03:51, 11.00s/it][2022-12-21 03:21:54,120] [INFO] [logging.py:68:log_dist] [Rank 0] step=5040, skipped=7, lr=[0.0], mom=[[0.9, 0.999]]
[2022-12-21 03:21:54,122] [INFO] [timer.py:197:stop] 0/10080, RunningAvgSamplesPerSec=5.881383155009212, CurrSamplesPerSec=5.364761942753, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4980/5000 [16:59:34<03:39, 10.99s/it][2022-12-21 03:22:05,171] [INFO] [timer.py:197:stop] 0/10082, RunningAvgSamplesPerSec=5.881378246182036, CurrSamplesPerSec=5.280441887068784, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4981/5000 [16:59:45<03:29, 11.00s/it][2022-12-21 03:22:16,174] [INFO] [timer.py:197:stop] 0/10084, RunningAvgSamplesPerSec=5.881378418990187, CurrSamplesPerSec=5.3117329230357155, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4982/5000 [16:59:56<03:17, 10.99s/it][2022-12-21 03:22:27,120] [INFO] [timer.py:197:stop] 0/10086, RunningAvgSamplesPerSec=5.881384629528429, CurrSamplesPerSec=5.320820669807597, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4983/5000 [17:00:07<03:07, 11.00s/it][2022-12-21 03:22:38,160] [INFO] [timer.py:197:stop] 0/10088, RunningAvgSamplesPerSec=5.881380819315018, CurrSamplesPerSec=5.3151060627528945, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4984/5000 [17:00:18<02:56, 11.01s/it][2022-12-21 03:22:49,214] [INFO] [timer.py:197:stop] 0/10090, RunningAvgSamplesPerSec=5.881375611525982, CurrSamplesPerSec=5.301347023906035, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4985/5000 [17:00:29<02:45, 11.03s/it][2022-12-21 03:23:00,239] [INFO] [timer.py:197:stop] 0/10092, RunningAvgSamplesPerSec=5.881373397196765, CurrSamplesPerSec=5.340788758161633, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4986/5000 [17:00:40<02:34, 11.03s/it][2022-12-21 03:23:11,310] [INFO] [timer.py:197:stop] 0/10094, RunningAvgSamplesPerSec=5.881366258632091, CurrSamplesPerSec=5.2960341849759835, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4987/5000 [17:00:51<02:23, 11.02s/it][2022-12-21 03:23:22,308] [INFO] [timer.py:197:stop] 0/10096, RunningAvgSamplesPerSec=5.881366958902828, CurrSamplesPerSec=5.312095357485689, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4988/5000 [17:01:02<02:12, 11.02s/it][2022-12-21 03:23:33,347] [INFO] [timer.py:197:stop] 0/10098, RunningAvgSamplesPerSec=5.881363276141535, CurrSamplesPerSec=5.297693329016797, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4989/5000 [17:01:13<02:01, 11.04s/it][2022-12-21 03:23:44,385] [INFO] [logging.py:68:log_dist] [Rank 0] step=5050, skipped=7, lr=[0.0], mom=[[0.9, 0.999]]
[2022-12-21 03:23:44,386] [INFO] [timer.py:197:stop] 0/10100, RunningAvgSamplesPerSec=5.881359542013666, CurrSamplesPerSec=5.325262222620043, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4990/5000 [17:01:24<01:50, 11.02s/it][2022-12-21 03:23:55,394] [INFO] [timer.py:197:stop] 0/10102, RunningAvgSamplesPerSec=5.8813591452139375, CurrSamplesPerSec=5.307864013243388, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4991/5000 [17:01:35<01:39, 11.02s/it][2022-12-21 03:24:06,354] [INFO] [timer.py:197:stop] 0/10104, RunningAvgSamplesPerSec=5.881364061779938, CurrSamplesPerSec=5.35067200467225, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4992/5000 [17:01:46<01:28, 11.02s/it][2022-12-21 03:24:17,429] [INFO] [timer.py:197:stop] 0/10106, RunningAvgSamplesPerSec=5.881356451745012, CurrSamplesPerSec=5.299697518849052, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4993/5000 [17:01:57<01:17, 11.03s/it][2022-12-21 03:24:28,444] [INFO] [timer.py:197:stop] 0/10108, RunningAvgSamplesPerSec=5.881355443884368, CurrSamplesPerSec=5.3328732781455175, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4994/5000 [17:02:09<01:06, 11.03s/it][2022-12-21 03:24:39,488] [INFO] [timer.py:197:stop] 0/10110, RunningAvgSamplesPerSec=5.881351212272025, CurrSamplesPerSec=5.315564320545756, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4995/5000 [17:02:19<00:55, 11.02s/it][2022-12-21 03:24:50,495] [INFO] [timer.py:197:stop] 0/10112, RunningAvgSamplesPerSec=5.881350984915537, CurrSamplesPerSec=5.308833127046617, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4996/5000 [17:02:31<00:44, 11.02s/it][2022-12-21 03:25:01,524] [INFO] [timer.py:197:stop] 0/10114, RunningAvgSamplesPerSec=5.8813482856803985, CurrSamplesPerSec=5.305472346229712, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4997/5000 [17:02:42<00:33, 11.04s/it][2022-12-21 03:25:12,609] [INFO] [timer.py:197:stop] 0/10116, RunningAvgSamplesPerSec=5.881339695009787, CurrSamplesPerSec=5.292267606688984, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4998/5000 [17:02:53<00:22, 11.05s/it][2022-12-21 03:25:23,644] [INFO] [timer.py:197:stop] 0/10118, RunningAvgSamplesPerSec=5.881336428470138, CurrSamplesPerSec=5.324769758921748, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|█████████▉| 4999/5000 [17:03:04<00:11, 11.04s/it][2022-12-21 03:25:34,660] [INFO] [logging.py:68:log_dist] [Rank 0] step=5060, skipped=7, lr=[0.0], mom=[[0.9, 0.999]]
[2022-12-21 03:25:34,661] [INFO] [timer.py:197:stop] 0/10120, RunningAvgSamplesPerSec=5.881335021976626, CurrSamplesPerSec=5.317251735366098, MemAllocated=3.0GB, MaxMemAllocated=19.53GB

100%|██████████| 5000/5000 [17:03:15<00:00, 11.02s/it]
                                                      
{'loss': 0.0001, 'learning_rate': 0.0, 'epoch': 121.94}

100%|██████████| 5000/5000 [17:03:15<00:00, 11.02s/it][INFO|trainer.py:2955] 2022-12-21 03:25:39,586 >> ***** Running Evaluation *****
[INFO|trainer.py:2957] 2022-12-21 03:25:39,586 >>   Num examples = 1325
[INFO|trainer.py:2960] 2022-12-21 03:25:39,586 >>   Batch size = 8


  0%|          | 0/166 [00:00<?, ?it/s]

  1%|          | 2/166 [00:05<07:53,  2.89s/it]

  2%|▏         | 3/166 [00:11<11:10,  4.11s/it]

  2%|▏         | 4/166 [00:17<12:56,  4.79s/it]

  3%|▎         | 5/166 [00:24<15:07,  5.64s/it]

  4%|▎         | 6/166 [00:32<16:32,  6.20s/it]

  4%|▍         | 7/166 [00:38<16:32,  6.24s/it]

  5%|▍         | 8/166 [00:44<16:08,  6.13s/it]

  5%|▌         | 9/166 [00:50<15:53,  6.07s/it]

  6%|▌         | 10/166 [00:56<15:33,  5.98s/it]

  7%|▋         | 11/166 [01:01<15:11,  5.88s/it]

  7%|▋         | 12/166 [01:07<15:10,  5.92s/it]

  8%|▊         | 13/166 [01:14<15:30,  6.08s/it]

  8%|▊         | 14/166 [01:21<16:03,  6.34s/it]

  9%|▉         | 15/166 [01:26<15:26,  6.13s/it]

 10%|▉         | 16/166 [01:34<16:13,  6.49s/it]

 10%|█         | 17/166 [01:40<16:16,  6.55s/it]

 11%|█         | 18/166 [01:47<15:56,  6.46s/it]

 11%|█▏        | 19/166 [01:52<15:24,  6.29s/it]

 12%|█▏        | 20/166 [02:03<18:33,  7.62s/it]

 13%|█▎        | 21/166 [02:10<18:07,  7.50s/it]

 13%|█▎        | 22/166 [02:18<17:52,  7.45s/it]

 14%|█▍        | 23/166 [02:25<17:50,  7.48s/it]

 14%|█▍        | 24/166 [02:32<16:59,  7.18s/it]

 15%|█▌        | 25/166 [02:39<16:53,  7.19s/it]

 16%|█▌        | 26/166 [02:46<16:40,  7.14s/it]

 16%|█▋        | 27/166 [02:53<16:42,  7.21s/it]

 17%|█▋        | 28/166 [02:59<15:46,  6.86s/it]

 17%|█▋        | 29/166 [03:08<16:34,  7.26s/it]

 18%|█▊        | 30/166 [03:14<16:11,  7.14s/it]

 19%|█▊        | 31/166 [03:22<16:09,  7.18s/it]

 19%|█▉        | 32/166 [03:29<16:22,  7.33s/it]

 20%|█▉        | 33/166 [03:37<16:16,  7.35s/it]

 20%|██        | 34/166 [03:43<15:32,  7.07s/it]

 21%|██        | 35/166 [03:51<16:00,  7.33s/it]

 22%|██▏       | 36/166 [03:59<15:59,  7.38s/it]

 22%|██▏       | 37/166 [04:05<15:18,  7.12s/it]

 23%|██▎       | 38/166 [04:14<16:05,  7.54s/it]

 23%|██▎       | 39/166 [04:21<15:31,  7.34s/it]

 24%|██▍       | 40/166 [04:28<15:40,  7.46s/it]

 25%|██▍       | 41/166 [04:35<15:04,  7.23s/it]

 25%|██▌       | 42/166 [04:41<14:28,  7.00s/it]

 26%|██▌       | 43/166 [04:48<13:57,  6.81s/it]

 27%|██▋       | 44/166 [04:56<14:53,  7.33s/it]

 27%|██▋       | 45/166 [05:04<15:13,  7.55s/it]

 28%|██▊       | 46/166 [05:14<16:37,  8.31s/it]

 28%|██▊       | 47/166 [05:23<16:25,  8.28s/it]

 29%|██▉       | 48/166 [05:30<15:34,  7.92s/it]

 30%|██▉       | 49/166 [05:36<14:14,  7.31s/it]

 30%|███       | 50/166 [05:44<14:50,  7.67s/it]

 31%|███       | 51/166 [05:52<14:34,  7.60s/it]

 31%|███▏      | 52/166 [05:59<14:27,  7.61s/it]

 32%|███▏      | 53/166 [06:06<14:04,  7.47s/it]

 33%|███▎      | 54/166 [06:14<14:12,  7.61s/it]

 33%|███▎      | 55/166 [06:20<13:08,  7.11s/it]

 34%|███▎      | 56/166 [06:26<12:29,  6.82s/it]

 34%|███▍      | 57/166 [06:35<13:06,  7.21s/it]

 35%|███▍      | 58/166 [06:40<12:05,  6.71s/it]

 36%|███▌      | 59/166 [06:47<12:10,  6.83s/it]

 36%|███▌      | 60/166 [06:54<12:13,  6.92s/it]

 37%|███▋      | 61/166 [07:03<13:15,  7.58s/it]

 37%|███▋      | 62/166 [07:11<13:00,  7.50s/it]

 38%|███▊      | 63/166 [07:18<12:45,  7.43s/it]

 39%|███▊      | 64/166 [07:27<13:27,  7.92s/it]

 39%|███▉      | 65/166 [07:36<13:35,  8.08s/it]

 40%|███▉      | 66/166 [07:42<12:51,  7.71s/it]

 40%|████      | 67/166 [07:49<12:20,  7.48s/it]

 41%|████      | 68/166 [07:54<11:03,  6.78s/it]

 42%|████▏     | 69/166 [08:01<10:59,  6.80s/it]

 42%|████▏     | 70/166 [08:09<11:29,  7.18s/it]

 43%|████▎     | 71/166 [08:17<11:42,  7.39s/it]

 43%|████▎     | 72/166 [08:24<11:10,  7.13s/it]

 44%|████▍     | 73/166 [08:30<10:43,  6.92s/it]

 45%|████▍     | 74/166 [08:37<10:19,  6.73s/it]

 45%|████▌     | 75/166 [08:43<10:01,  6.61s/it]

 46%|████▌     | 76/166 [08:50<10:13,  6.82s/it]

 46%|████▋     | 77/166 [08:56<09:38,  6.50s/it]

 47%|████▋     | 78/166 [09:03<09:47,  6.68s/it]

 48%|████▊     | 79/166 [09:11<10:14,  7.06s/it]

 48%|████▊     | 80/166 [09:19<10:35,  7.38s/it]

 49%|████▉     | 81/166 [09:27<10:40,  7.53s/it]

 49%|████▉     | 82/166 [09:34<10:17,  7.35s/it]

 50%|█████     | 83/166 [09:39<09:20,  6.75s/it]

 51%|█████     | 84/166 [09:46<09:05,  6.65s/it]

 51%|█████     | 85/166 [09:53<09:13,  6.84s/it]

 52%|█████▏    | 86/166 [10:01<09:41,  7.27s/it]

 52%|█████▏    | 87/166 [10:09<09:41,  7.36s/it]

 53%|█████▎    | 88/166 [10:15<09:18,  7.16s/it]

 54%|█████▎    | 89/166 [10:23<09:15,  7.21s/it]

 54%|█████▍    | 90/166 [10:29<08:52,  7.01s/it]

 55%|█████▍    | 91/166 [10:36<08:38,  6.92s/it]

 55%|█████▌    | 92/166 [10:42<08:12,  6.66s/it]

 56%|█████▌    | 93/166 [10:50<08:37,  7.08s/it]

 57%|█████▋    | 94/166 [11:00<09:36,  8.01s/it]

 57%|█████▋    | 95/166 [11:06<08:30,  7.18s/it]

 58%|█████▊    | 96/166 [11:12<08:09,  6.99s/it]

 58%|█████▊    | 97/166 [11:22<09:09,  7.97s/it]

 59%|█████▉    | 98/166 [11:32<09:41,  8.55s/it]

 60%|█████▉    | 99/166 [11:40<09:10,  8.22s/it]

 60%|██████    | 100/166 [11:46<08:22,  7.62s/it]

 61%|██████    | 101/166 [11:53<07:55,  7.31s/it]

 61%|██████▏   | 102/166 [12:01<08:17,  7.78s/it]

 62%|██████▏   | 103/166 [12:10<08:31,  8.12s/it]

 63%|██████▎   | 104/166 [12:18<08:07,  7.87s/it]

 63%|██████▎   | 105/166 [12:26<08:06,  7.97s/it]

 64%|██████▍   | 106/166 [12:36<08:31,  8.52s/it]

 64%|██████▍   | 107/166 [12:45<08:41,  8.84s/it]

 65%|██████▌   | 108/166 [12:51<07:32,  7.81s/it]

 66%|██████▌   | 109/166 [12:57<07:00,  7.37s/it]

 66%|██████▋   | 110/166 [13:04<06:54,  7.40s/it]

 67%|██████▋   | 111/166 [13:10<06:23,  6.97s/it]

 67%|██████▋   | 112/166 [13:18<06:22,  7.08s/it]

 68%|██████▊   | 113/166 [13:26<06:38,  7.52s/it]

 69%|██████▊   | 114/166 [13:34<06:29,  7.50s/it]

 69%|██████▉   | 115/166 [13:40<06:04,  7.14s/it]

 70%|██████▉   | 116/166 [13:46<05:43,  6.88s/it]

 70%|███████   | 117/166 [13:54<05:46,  7.07s/it]

 71%|███████   | 118/166 [14:02<05:50,  7.30s/it]

 72%|███████▏  | 119/166 [14:09<05:48,  7.42s/it]

 72%|███████▏  | 120/166 [14:18<05:52,  7.65s/it]

 73%|███████▎  | 121/166 [14:26<05:53,  7.86s/it]

 73%|███████▎  | 122/166 [14:34<05:42,  7.79s/it]

 74%|███████▍  | 123/166 [14:41<05:29,  7.65s/it]

 75%|███████▍  | 124/166 [14:48<05:10,  7.39s/it]

 75%|███████▌  | 125/166 [14:55<05:03,  7.40s/it]

 76%|███████▌  | 126/166 [15:04<05:08,  7.72s/it]

 77%|███████▋  | 127/166 [15:09<04:29,  6.91s/it]

 77%|███████▋  | 128/166 [15:19<04:58,  7.86s/it]

 78%|███████▊  | 129/166 [15:27<04:51,  7.87s/it]

 78%|███████▊  | 130/166 [15:33<04:32,  7.57s/it]

 79%|███████▉  | 131/166 [15:40<04:19,  7.41s/it]

 80%|███████▉  | 132/166 [15:49<04:18,  7.61s/it]

 80%|████████  | 133/166 [15:56<04:12,  7.65s/it]

 81%|████████  | 134/166 [16:03<03:54,  7.34s/it]

 81%|████████▏ | 135/166 [16:09<03:36,  6.98s/it]

 82%|████████▏ | 136/166 [16:19<04:00,  8.02s/it]

 83%|████████▎ | 137/166 [16:28<04:00,  8.31s/it]

 83%|████████▎ | 138/166 [16:39<04:07,  8.84s/it]

 84%|████████▎ | 139/166 [16:47<03:54,  8.67s/it]

 84%|████████▍ | 140/166 [16:53<03:26,  7.96s/it]

 85%|████████▍ | 141/166 [16:59<03:03,  7.35s/it]

 86%|████████▌ | 142/166 [17:06<02:54,  7.29s/it]

 86%|████████▌ | 143/166 [17:12<02:35,  6.75s/it]

 87%|████████▋ | 144/166 [17:20<02:41,  7.36s/it]

 87%|████████▋ | 145/166 [17:30<02:49,  8.09s/it]

 88%|████████▊ | 146/166 [17:40<02:49,  8.46s/it]

 89%|████████▊ | 147/166 [17:48<02:40,  8.46s/it]

 89%|████████▉ | 148/166 [17:56<02:31,  8.44s/it]

 90%|████████▉ | 149/166 [18:06<02:27,  8.66s/it]

 90%|█████████ | 150/166 [18:15<02:23,  8.96s/it]

 91%|█████████ | 151/166 [18:23<02:10,  8.68s/it]

 92%|█████████▏| 152/166 [18:29<01:49,  7.80s/it]

 92%|█████████▏| 153/166 [18:36<01:36,  7.40s/it]

 93%|█████████▎| 154/166 [18:43<01:28,  7.38s/it]

 93%|█████████▎| 155/166 [18:51<01:22,  7.49s/it]

 94%|█████████▍| 156/166 [18:56<01:10,  7.00s/it]

 95%|█████████▍| 157/166 [19:03<01:02,  6.91s/it]

 95%|█████████▌| 158/166 [19:09<00:52,  6.52s/it]

 96%|█████████▌| 159/166 [19:14<00:41,  5.98s/it]

 96%|█████████▋| 160/166 [19:19<00:34,  5.70s/it]

 97%|█████████▋| 161/166 [19:25<00:29,  5.82s/it]

 98%|█████████▊| 162/166 [19:32<00:24,  6.15s/it]

 98%|█████████▊| 163/166 [19:39<00:19,  6.46s/it]

 99%|█████████▉| 164/166 [19:45<00:12,  6.37s/it]

 99%|█████████▉| 165/166 [19:51<00:06,  6.24s/it]

100%|██████████| 166/166 [19:55<00:00,  5.63s/it]
                                                      

                                                 
{'eval_loss': 0.291259765625, 'eval_wer': 8.473140457818456, 'eval_runtime': 1274.9287, 'eval_samples_per_second': 1.039, 'eval_steps_per_second': 0.13, 'epoch': 121.94}

100%|██████████| 5000/5000 [17:24:30<00:00, 11.02s/it]

100%|██████████| 166/166 [21:09<00:00,  5.63s/it]

                                                 [INFO|trainer.py:2700] 2022-12-21 03:46:54,517 >> Saving model checkpoint to ./checkpoint-5000
[INFO|configuration_utils.py:447] 2022-12-21 03:46:54,518 >> Configuration saved in ./checkpoint-5000/config.json
[INFO|modeling_utils.py:1680] 2022-12-21 03:46:58,246 >> Model weights saved in ./checkpoint-5000/pytorch_model.bin
[INFO|feature_extraction_utils.py:368] 2022-12-21 03:46:58,261 >> Feature extractor saved in ./checkpoint-5000/preprocessor_config.json
[2022-12-21 03:46:58,261] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step5060 is begin to save!
/home/milan/hf_env/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
  warnings.warn(
[2022-12-21 03:46:58,272] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: ./checkpoint-5000/global_step5060/mp_rank_00_model_states.pt
[2022-12-21 03:46:58,272] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-5000/global_step5060/mp_rank_00_model_states.pt...
[2022-12-21 03:47:01,980] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-5000/global_step5060/mp_rank_00_model_states.pt.
[2022-12-21 03:47:01,981] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving ./checkpoint-5000/global_step5060/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2022-12-21 03:47:16,832] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved ./checkpoint-5000/global_step5060/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2022-12-21 03:47:16,833] [INFO] [engine.py:3269:_save_zero_checkpoint] zero checkpoint saved ./checkpoint-5000/global_step5060/zero_pp_rank_0_mp_rank_00_optim_states.pt
[2022-12-21 03:47:16,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5060 is ready now!
[INFO|feature_extraction_utils.py:368] 2022-12-21 03:47:22,152 >> Feature extractor saved in ./preprocessor_config.json
[INFO|trainer.py:1892] 2022-12-21 03:49:32,139 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)


[INFO|trainer.py:2016] 2022-12-21 03:49:32,139 >> Loading best model from ./checkpoint-2000 (score: 8.37737794884072).
[2022-12-21 03:49:32,144] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.7, git-hash=unknown, git-branch=unknown
[2022-12-21 03:49:32,204] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2022-12-21 03:49:33,733] [WARNING] [cpu_adam.py:83:__init__] FP16 params for CPUAdam may not work on AMD CPUs
Installed CUDA version 11.6 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Using /home/milan/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module cpu_adam, skipping build step...
Loading extension module cpu_adam...
Time to load cpu_adam op: 3.4642910957336426 seconds
Adam Optimizer #1 is created with AVX2 arithmetic capability.
Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1
[2022-12-21 03:49:37,991] [INFO] [logging.py:68:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2022-12-21 03:49:38,296] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2022-12-21 03:49:38,296] [INFO] [utils.py:52:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2022-12-21 03:49:38,297] [INFO] [logging.py:68:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
[2022-12-21 03:49:38,297] [INFO] [stage_1_and_2.py:140:__init__] Reduce bucket size 200000000
[2022-12-21 03:49:38,297] [INFO] [stage_1_and_2.py:141:__init__] Allgather bucket size 200000000
[2022-12-21 03:49:38,297] [INFO] [stage_1_and_2.py:142:__init__] CPU Offload: True
[2022-12-21 03:49:38,297] [INFO] [stage_1_and_2.py:143:__init__] Round robin gradient partitioning: False
Using /home/milan/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00044226646423339844 seconds
Rank: 0 partition count [1] and sizes[(1543304960, False)] 
[2022-12-21 03:49:42,600] [INFO] [utils.py:827:see_memory_usage] Before initializing optimizer states
[2022-12-21 03:49:42,600] [INFO] [utils.py:828:see_memory_usage] MA 6.0 GB         Max_MA 19.53 GB         CA 29.61 GB         Max_CA 30 GB 
[2022-12-21 03:49:42,600] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 48.17 GB, percent = 24.5%
[2022-12-21 03:49:49,256] [INFO] [utils.py:827:see_memory_usage] After initializing optimizer states
[2022-12-21 03:49:49,257] [INFO] [utils.py:828:see_memory_usage] MA 6.0 GB         Max_MA 6.0 GB         CA 29.61 GB         Max_CA 30 GB 
[2022-12-21 03:49:49,257] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 68.2 GB, percent = 34.7%
[2022-12-21 03:49:49,257] [INFO] [stage_1_and_2.py:525:__init__] optimizer state initialized
[2022-12-21 03:49:49,336] [INFO] [utils.py:827:see_memory_usage] After initializing ZeRO optimizer
[2022-12-21 03:49:49,336] [INFO] [utils.py:828:see_memory_usage] MA 6.0 GB         Max_MA 6.0 GB         CA 29.61 GB         Max_CA 30 GB 
[2022-12-21 03:49:49,336] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 68.2 GB, percent = 34.7%
[2022-12-21 03:49:49,366] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2022-12-21 03:49:49,366] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupDecayLR
[2022-12-21 03:49:49,366] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupDecayLR object at 0x7ff9683c0550>
[2022-12-21 03:49:49,366] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05], mom=[[0.9, 0.999]]
[2022-12-21 03:49:49,368] [INFO] [config.py:1020:print] DeepSpeedEngine configuration:
[2022-12-21 03:49:49,368] [INFO] [config.py:1024:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2022-12-21 03:49:49,368] [INFO] [config.py:1024:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2022-12-21 03:49:49,368] [INFO] [config.py:1024:print]   amp_enabled .................. False
[2022-12-21 03:49:49,368] [INFO] [config.py:1024:print]   amp_params ................... False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   bfloat16_enabled ............. False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   checkpoint_parallel_write_pipeline  False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   checkpoint_tag_validation_enabled  True
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   checkpoint_tag_validation_fail  False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7ff967e80df0>
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   communication_data_type ...... None
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   curriculum_enabled ........... False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   curriculum_params ............ False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   dataloader_drop_last ......... False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   disable_allgather ............ False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   dump_state ................... False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   dynamic_loss_scale_args ...... {'init_scale': 65536, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   eigenvalue_enabled ........... False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   eigenvalue_gas_boundary_resolution  1
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   eigenvalue_layer_num ......... 0
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   eigenvalue_max_iter .......... 100
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   eigenvalue_stability ......... 1e-06
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   eigenvalue_tol ............... 0.01
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   eigenvalue_verbose ........... False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   elasticity_enabled ........... False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   flops_profiler_config ........ {
    "enabled": false, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   fp16_auto_cast ............... False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   fp16_enabled ................. True
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   fp16_master_weights_and_gradients  False
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   global_rank .................. 0
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   grad_accum_dtype ............. None
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   gradient_accumulation_steps .. 2
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   gradient_clipping ............ 1.0
[2022-12-21 03:49:49,369] [INFO] [config.py:1024:print]   gradient_predivide_factor .... 1.0
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   initial_dynamic_scale ........ 65536
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   load_universal_checkpoint .... False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   loss_scale ................... 0
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   memory_breakdown ............. False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   monitor_config ............... <deepspeed.monitor.config.DeepSpeedMonitorConfig object at 0x7ff967e80f40>
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   optimizer_legacy_fusion ...... False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   optimizer_name ............... adamw
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   optimizer_params ............. {'lr': 1e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   pld_enabled .................. False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   pld_params ................... False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   prescale_gradients ........... False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   scheduler_name ............... WarmupDecayLR
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   scheduler_params ............. {'last_batch_iteration': -1, 'total_num_steps': 5000, 'warmup_min_lr': 0, 'warmup_max_lr': 1e-05, 'warmup_num_steps': 500}
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   sparse_attention ............. None
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   sparse_gradients_enabled ..... False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   steps_per_print .............. 10
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   train_batch_size ............. 64
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   train_micro_batch_size_per_gpu  32
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   use_node_local_storage ....... False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   wall_clock_breakdown ......... False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   world_size ................... 1
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   zero_allow_untested_optimizer  False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   zero_enabled ................. True
[2022-12-21 03:49:49,370] [INFO] [config.py:1024:print]   zero_optimization_stage ...... 2
[2022-12-21 03:49:49,370] [INFO] [config.py:1009:print_user_config]   json = {
    "fp16": {
        "enabled": true, 
        "loss_scale": 0, 
        "loss_scale_window": 1000, 
        "initial_scale_power": 16, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "optimizer": {
        "type": "AdamW", 
        "params": {
            "lr": 1e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.0
        }
    }, 
    "scheduler": {
        "type": "WarmupDecayLR", 
        "params": {
            "last_batch_iteration": -1, 
            "total_num_steps": 5.000000e+03, 
            "warmup_min_lr": 0, 
            "warmup_max_lr": 1e-05, 
            "warmup_num_steps": 500
        }
    }, 
    "zero_optimization": {
        "stage": 2, 
        "offload_optimizer": {
            "device": "cpu", 
            "pin_memory": true
        }, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 2.000000e+08, 
        "overlap_comm": true, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 2.000000e+08, 
        "contiguous_gradients": true
    }, 
    "gradient_accumulation_steps": 2, 
    "gradient_clipping": 1.0, 
    "train_batch_size": 64, 
    "train_micro_batch_size_per_gpu": 32
}
Using /home/milan/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0002999305725097656 seconds
[INFO|deepspeed.py:357] 2022-12-21 03:49:49,371 >> Attempting to resume from ./checkpoint-2000
[2022-12-21 03:49:49,372] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from ./checkpoint-2000/global_step2024/mp_rank_00_model_states.pt...
[2022-12-21 03:49:51,318] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from ./checkpoint-2000/global_step2024/mp_rank_00_model_states.pt.
[2022-12-21 03:49:51,465] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from ./checkpoint-2000/global_step2024/mp_rank_00_model_states.pt...
[2022-12-21 03:49:52,499] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from ./checkpoint-2000/global_step2024/mp_rank_00_model_states.pt.
[2022-12-21 03:49:52,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from ./checkpoint-2000/global_step2024/zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2022-12-21 03:50:04,455] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from ./checkpoint-2000/global_step2024/zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2022-12-21 03:50:04,455] [INFO] [engine.py:2900:_get_all_zero_checkpoint_state_dicts] successfully read 1 ZeRO state_dicts for rank 0
[2022-12-21 03:50:07,602] [INFO] [engine.py:2840:_load_zero_checkpoint] loading 1 zero partition checkpoints for rank 0

                                                      
{'train_runtime': 62864.0618, 'train_samples_per_second': 5.09, 'train_steps_per_second': 0.08, 'train_loss': 0.00465274755358696, 'epoch': 121.94}

100%|██████████| 5000/5000 [17:27:44<00:00, 11.02s/it]
100%|██████████| 5000/5000 [17:27:44<00:00, 12.57s/it]
[INFO|trainer.py:2700] 2022-12-21 03:50:08,457 >> Saving model checkpoint to ./
[INFO|configuration_utils.py:447] 2022-12-21 03:50:08,458 >> Configuration saved in ./config.json
[INFO|modeling_utils.py:1680] 2022-12-21 03:50:12,947 >> Model weights saved in ./pytorch_model.bin
[INFO|feature_extraction_utils.py:368] 2022-12-21 03:50:12,962 >> Feature extractor saved in ./preprocessor_config.json
[INFO|trainer.py:2700] 2022-12-21 03:50:12,963 >> Saving model checkpoint to ./
[INFO|configuration_utils.py:447] 2022-12-21 03:50:12,963 >> Configuration saved in ./config.json
[INFO|modeling_utils.py:1680] 2022-12-21 03:50:17,513 >> Model weights saved in ./pytorch_model.bin
[INFO|feature_extraction_utils.py:368] 2022-12-21 03:50:17,528 >> Feature extractor saved in ./preprocessor_config.json