[2023-12-17:11:19:23:INFO:sockeye.utils:log_sockeye_version] Sockeye: 3.1.36, commit unknown, path /home/amoryo/sign-language/signbank-annotation/signbank-plus/signbank_plus/nmt/sockeye/sockeye/sockeye/__init__.py
[2023-12-17:11:19:23:INFO:sockeye.utils:log_torch_version] PyTorch: 1.12.1+cu113 (/home/amoryo/.local/lib/python3.10/site-packages/torch/__init__.py)
[2023-12-17:11:19:23:INFO:sockeye.utils:log_basic_info] Command: /home/amoryo/sign-language/signbank-annotation/signbank-plus/signbank_plus/nmt/sockeye/sockeye/sockeye/train.py -d /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data --weight-tying-type none --batch-size 512 --max-seq-len 512:128 --decode-and-evaluate 500 --validation-source /home/amoryo/sign-language/signbank-annotation/signbank-plus/data/parallel/cleaned/dev.source.tokenized --validation-target /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/dev.target.bpe --optimized-metric chrf --max-num-checkpoint-not-improved 20 --output /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model --params /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded/model/params.best
[2023-12-17:11:19:23:INFO:sockeye.utils:log_basic_info] Arguments: Namespace(config=None, source=None, source_factors=[], source_factors_use_source_vocab=[], target_factors=[], target_factors_use_target_vocab=[], target=None, end_of_prepending_tag=None, prepared_data='/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data', validation_source='/home/amoryo/sign-language/signbank-annotation/signbank-plus/data/parallel/cleaned/dev.source.tokenized', validation_source_factors=[], validation_target='/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/dev.target.bpe', validation_target_factors=[], no_bucketing=False, bucket_width=8, bucket_scaling=False, max_seq_len=(512, 128), source_vocab=None, target_vocab=None, source_factor_vocabs=[], target_factor_vocabs=[], shared_vocab=False, num_words=(0, 0), word_min_count=(1, 1), pad_vocab_to_multiple_of=8, output='/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model', overwrite_output=False, params='/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded/model/params.best', allow_missing_params=False, ignore_extra_params=False, encoder='transformer', decoder='transformer', num_layers=(6, 6), transformer_model_size=(512, 512), transformer_attention_heads=(8, 8), transformer_feed_forward_num_hidden=(2048, 2048), transformer_feed_forward_use_glu=False, transformer_activation_type=('relu', 'relu'), transformer_positional_embedding_type='fixed', transformer_block_prepended_cross_attention=False, transformer_preprocess=('n', 'n'), transformer_postprocess=('dr', 'dr'), lhuc=None, num_embed=(None, None), source_factors_num_embed=[], target_factors_num_embed=[], source_factors_combine=[], target_factors_combine=[], source_factors_share_embedding=[], target_factors_share_embedding=[], weight_tying_type='none', dtype='float32', clamp_to_dtype=False, amp=False, apex_amp=False, neural_vocab_selection=None, neural_vocab_selection_block_loss=False, batch_size=512, batch_type='word', batch_sentences_multiple_of=8, update_interval=1, label_smoothing=0.1, label_smoothing_impl='mxnet', length_task=None, length_task_weight=1.0, length_task_layers=1, bow_task_weight=1.0, bow_task_pos_weight=10, target_factors_weight=[1.0], optimized_metric='chrf', checkpoint_interval=4000, min_samples=None, max_samples=None, min_updates=None, max_updates=None, max_seconds=None, max_checkpoints=None, max_num_checkpoint_not_improved=20, checkpoint_improvement_threshold=0.0, min_num_epochs=None, max_num_epochs=None, embed_dropout=(0.0, 0.0), transformer_dropout_attention=(0.1, 0.1), transformer_dropout_act=(0.1, 0.1), transformer_dropout_prepost=(0.1, 0.1), optimizer='adam', optimizer_betas=(0.9, 0.999), optimizer_eps=1e-08, dist=False, initial_learning_rate=0.0002, weight_decay=0.0, momentum=0.0, gradient_clipping_threshold=1.0, gradient_clipping_type='none', learning_rate_scheduler_type='plateau-reduce', learning_rate_reduce_factor=0.9, learning_rate_reduce_num_not_improved=8, learning_rate_warmup=0, no_reload_on_learning_rate_reduce=False, fixed_param_strategy=None, fixed_param_names=[], local_rank=None, deepspeed_fp16=False, deepspeed_bf16=False, decode_and_evaluate=500, stop_training_on_decoder_failure=False, seed=1, keep_last_params=-1, keep_initializations=False, cache_last_best_params=0, cache_strategy='best', cache_metric='perplexity', dry_run=False, device_id=0, use_cpu=False, env=None, tf32=True, quiet=False, quiet_secondary_workers=False, no_logfile=False, loglevel='INFO', loglevel_secondary_workers='INFO')
[2023-12-17:11:19:23:INFO:__main__:train] Adjusting maximum length to reserve space for a BOS/EOS marker. New maximum length: (513, 129)
[2023-12-17:11:19:24:INFO:sockeye.utils:init_device] CUDA: allow tf32 (float32 but with 10 bits precision)
[2023-12-17:11:19:24:INFO:__main__:train] Training Device: cuda:0
[2023-12-17:11:19:24:INFO:sockeye.utils:seed_rngs] Random seed: 1
[2023-12-17:11:19:24:INFO:sockeye.utils:seed_rngs] PyTorch seed: 1
[2023-12-17:11:19:24:INFO:sockeye.data_io:get_prepared_data_iters] ===============================
[2023-12-17:11:19:24:INFO:sockeye.data_io:get_prepared_data_iters] Creating training data iterator
[2023-12-17:11:19:24:INFO:sockeye.data_io:get_prepared_data_iters] ===============================
[2023-12-17:11:19:24:INFO:sockeye.vocab:vocab_from_json] Vocabulary (1232 words) loaded from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data/vocab.src.0.json"
[2023-12-17:11:19:24:INFO:sockeye.vocab:vocab_from_json] Vocabulary (5976 words) loaded from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data/vocab.trg.0.json"
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] Tokens: source 15626084 target 2020240
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] Number of <unk> tokens: source 1 target 116
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] Vocabulary coverage: source 100% target 100%
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] 354707 sequences across 65 buckets
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] 20 sequences did not fit into buckets and were discarded
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (8, 8): 1 samples in 1 batches of 256, ~512.0 target tokens/batch, trg/src length ratio: 0.33 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (16, 16): 29122 samples in 228 batches of 128, ~523.0 target tokens/batch, trg/src length ratio: 0.28 (+-0.18)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (24, 24): 57389 samples in 513 batches of 112, ~509.1 target tokens/batch, trg/src length ratio: 0.22 (+-0.13)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (32, 32): 123411 samples in 1102 batches of 112, ~516.0 target tokens/batch, trg/src length ratio: 0.17 (+-0.10)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (40, 40): 36766 samples in 354 batches of 104, ~507.4 target tokens/batch, trg/src length ratio: 0.14 (+-0.12)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (48, 48): 44876 samples in 432 batches of 104, ~518.2 target tokens/batch, trg/src length ratio: 0.12 (+-0.12)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (56, 56): 21849 samples in 228 batches of 96, ~506.0 target tokens/batch, trg/src length ratio: 0.10 (+-0.11)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (64, 64): 6758 samples in 77 batches of 88, ~510.7 target tokens/batch, trg/src length ratio: 0.10 (+-0.13)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (72, 72): 8500 samples in 97 batches of 88, ~530.5 target tokens/batch, trg/src length ratio: 0.09 (+-0.16)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (80, 80): 2765 samples in 39 batches of 72, ~492.1 target tokens/batch, trg/src length ratio: 0.10 (+-0.26)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (88, 88): 3715 samples in 59 batches of 64, ~499.4 target tokens/batch, trg/src length ratio: 0.10 (+-0.20)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (96, 96): 2686 samples in 84 batches of 32, ~481.5 target tokens/batch, trg/src length ratio: 0.17 (+-0.26)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (104, 104): 722 samples in 13 batches of 56, ~484.0 target tokens/batch, trg/src length ratio: 0.11 (+-0.32)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (112, 112): 752 samples in 14 batches of 56, ~483.9 target tokens/batch, trg/src length ratio: 0.10 (+-0.31)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (120, 120): 417 samples in 9 batches of 48, ~512.6 target tokens/batch, trg/src length ratio: 0.13 (+-0.39)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (128, 128): 491 samples in 9 batches of 56, ~511.5 target tokens/batch, trg/src length ratio: 0.09 (+-0.31)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (136, 129): 402 samples in 8 batches of 56, ~521.1 target tokens/batch, trg/src length ratio: 0.07 (+-0.06)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (144, 129): 347 samples in 7 batches of 56, ~502.5 target tokens/batch, trg/src length ratio: 0.06 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (152, 129): 249 samples in 6 batches of 48, ~526.7 target tokens/batch, trg/src length ratio: 0.07 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (160, 129): 196 samples in 5 batches of 40, ~538.6 target tokens/batch, trg/src length ratio: 0.09 (+-0.06)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (168, 129): 297 samples in 8 batches of 40, ~529.3 target tokens/batch, trg/src length ratio: 0.08 (+-0.06)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (176, 129): 376 samples in 8 batches of 48, ~543.6 target tokens/batch, trg/src length ratio: 0.07 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (184, 129): 262 samples in 7 batches of 40, ~559.4 target tokens/batch, trg/src length ratio: 0.08 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (192, 129): 246 samples in 8 batches of 32, ~516.6 target tokens/batch, trg/src length ratio: 0.09 (+-0.06)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (200, 129): 213 samples in 7 batches of 32, ~531.5 target tokens/batch, trg/src length ratio: 0.08 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (208, 129): 269 samples in 9 batches of 32, ~525.7 target tokens/batch, trg/src length ratio: 0.08 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (216, 129): 331 samples in 11 batches of 32, ~533.0 target tokens/batch, trg/src length ratio: 0.08 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (224, 129): 317 samples in 14 batches of 24, ~462.2 target tokens/batch, trg/src length ratio: 0.09 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (232, 129): 350 samples in 15 batches of 24, ~518.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (240, 129): 356 samples in 15 batches of 24, ~519.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (248, 129): 368 samples in 16 batches of 24, ~502.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (256, 129): 451 samples in 19 batches of 24, ~551.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (264, 129): 454 samples in 19 batches of 24, ~567.3 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (272, 129): 497 samples in 21 batches of 24, ~580.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (280, 129): 483 samples in 21 batches of 24, ~608.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (288, 129): 519 samples in 33 batches of 16, ~415.3 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (296, 129): 572 samples in 36 batches of 16, ~416.2 target tokens/batch, trg/src length ratio: 0.09 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (304, 129): 500 samples in 32 batches of 16, ~435.5 target tokens/batch, trg/src length ratio: 0.09 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (312, 129): 550 samples in 35 batches of 16, ~431.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (320, 129): 608 samples in 38 batches of 16, ~436.8 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (328, 129): 561 samples in 36 batches of 16, ~448.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (336, 129): 537 samples in 34 batches of 16, ~480.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (344, 129): 481 samples in 31 batches of 16, ~479.5 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (352, 129): 568 samples in 36 batches of 16, ~495.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (360, 129): 516 samples in 33 batches of 16, ~487.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (368, 129): 468 samples in 30 batches of 16, ~488.4 target tokens/batch, trg/src length ratio: 0.08 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (376, 129): 554 samples in 35 batches of 16, ~503.9 target tokens/batch, trg/src length ratio: 0.08 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (384, 129): 512 samples in 32 batches of 16, ~520.2 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (392, 129): 486 samples in 31 batches of 16, ~522.2 target tokens/batch, trg/src length ratio: 0.08 (+-0.02)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (400, 129): 455 samples in 29 batches of 16, ~551.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (408, 129): 130 samples in 9 batches of 16, ~521.7 target tokens/batch, trg/src length ratio: 0.08 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (416, 129): 2 samples in 1 batches of 96, ~528.0 target tokens/batch, trg/src length ratio: 0.01 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (424, 129): 4 samples in 1 batches of 8, ~344.0 target tokens/batch, trg/src length ratio: 0.10 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:_load_shard] Loading shard /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data/shard.00000.
[2023-12-17:11:19:24:INFO:sockeye.data_io:get_validation_data_iter] =================================
[2023-12-17:11:19:24:INFO:sockeye.data_io:get_validation_data_iter] Creating validation data iterator
[2023-12-17:11:19:24:INFO:sockeye.data_io:get_validation_data_iter] =================================
[2023-12-17:11:19:24:INFO:sockeye.data_io:analyze_sequence_lengths] 2845 sequences of maximum length (513, 129) in '/home/amoryo/sign-language/signbank-annotation/signbank-plus/data/parallel/cleaned/dev.source.tokenized' and '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/dev.target.bpe'.
[2023-12-17:11:19:24:INFO:sockeye.data_io:analyze_sequence_lengths] Mean training target/source length ratio: 0.16 (+-0.12)
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] Tokens: source 128497 target 16716
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] Number of <unk> tokens: source 0 target 1
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] Vocabulary coverage: source 100% target 100%
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] 2845 sequences across 65 buckets
[2023-12-17:11:19:24:INFO:sockeye.data_io:log] 0 sequences did not fit into buckets and were discarded
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (16, 16): 242 samples in 2 batches of 128, ~523.0 target tokens/batch, trg/src length ratio: 0.29 (+-0.15)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (24, 24): 472 samples in 5 batches of 112, ~509.1 target tokens/batch, trg/src length ratio: 0.22 (+-0.11)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (32, 32): 920 samples in 9 batches of 112, ~516.0 target tokens/batch, trg/src length ratio: 0.17 (+-0.10)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (40, 40): 333 samples in 4 batches of 104, ~507.4 target tokens/batch, trg/src length ratio: 0.13 (+-0.07)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (48, 48): 382 samples in 4 batches of 104, ~518.2 target tokens/batch, trg/src length ratio: 0.12 (+-0.09)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (56, 56): 178 samples in 2 batches of 96, ~506.0 target tokens/batch, trg/src length ratio: 0.11 (+-0.15)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (64, 64): 63 samples in 1 batches of 88, ~510.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (72, 72): 50 samples in 1 batches of 88, ~530.5 target tokens/batch, trg/src length ratio: 0.08 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (80, 80): 18 samples in 1 batches of 72, ~492.1 target tokens/batch, trg/src length ratio: 0.11 (+-0.16)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (88, 88): 18 samples in 1 batches of 64, ~499.4 target tokens/batch, trg/src length ratio: 0.14 (+-0.20)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (96, 96): 20 samples in 1 batches of 32, ~481.5 target tokens/batch, trg/src length ratio: 0.21 (+-0.17)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (104, 104): 4 samples in 1 batches of 56, ~484.0 target tokens/batch, trg/src length ratio: 0.08 (+-0.02)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (112, 112): 10 samples in 1 batches of 56, ~483.9 target tokens/batch, trg/src length ratio: 0.07 (+-0.02)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (120, 120): 2 samples in 1 batches of 48, ~512.6 target tokens/batch, trg/src length ratio: 0.05 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (128, 128): 4 samples in 1 batches of 56, ~511.5 target tokens/batch, trg/src length ratio: 0.12 (+-0.08)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (136, 129): 1 samples in 1 batches of 56, ~521.1 target tokens/batch, trg/src length ratio: 0.10 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (144, 129): 2 samples in 1 batches of 56, ~502.5 target tokens/batch, trg/src length ratio: 0.17 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (152, 129): 2 samples in 1 batches of 48, ~526.7 target tokens/batch, trg/src length ratio: 0.15 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (160, 129): 2 samples in 1 batches of 40, ~538.6 target tokens/batch, trg/src length ratio: 0.13 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (176, 129): 5 samples in 1 batches of 48, ~543.6 target tokens/batch, trg/src length ratio: 0.05 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (184, 129): 4 samples in 1 batches of 40, ~559.4 target tokens/batch, trg/src length ratio: 0.08 (+-0.06)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (208, 129): 4 samples in 1 batches of 32, ~525.7 target tokens/batch, trg/src length ratio: 0.04 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (224, 129): 2 samples in 1 batches of 24, ~462.2 target tokens/batch, trg/src length ratio: 0.02 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (232, 129): 8 samples in 1 batches of 24, ~518.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (240, 129): 3 samples in 1 batches of 24, ~519.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.03)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (248, 129): 5 samples in 1 batches of 24, ~502.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (256, 129): 5 samples in 1 batches of 24, ~551.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (264, 129): 8 samples in 1 batches of 24, ~567.3 target tokens/batch, trg/src length ratio: 0.09 (+-0.02)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (272, 129): 4 samples in 1 batches of 24, ~580.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (280, 129): 3 samples in 1 batches of 24, ~608.7 target tokens/batch, trg/src length ratio: 0.07 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (288, 129): 5 samples in 1 batches of 16, ~415.3 target tokens/batch, trg/src length ratio: 0.02 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (296, 129): 4 samples in 1 batches of 16, ~416.2 target tokens/batch, trg/src length ratio: 0.08 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (304, 129): 3 samples in 1 batches of 16, ~435.5 target tokens/batch, trg/src length ratio: 0.12 (+-0.05)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (312, 129): 11 samples in 1 batches of 16, ~431.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.02)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (320, 129): 6 samples in 1 batches of 16, ~436.8 target tokens/batch, trg/src length ratio: 0.08 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (328, 129): 6 samples in 1 batches of 16, ~448.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (344, 129): 8 samples in 1 batches of 16, ~479.5 target tokens/batch, trg/src length ratio: 0.10 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (352, 129): 4 samples in 1 batches of 16, ~495.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (360, 129): 4 samples in 1 batches of 16, ~487.7 target tokens/batch, trg/src length ratio: 0.08 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (368, 129): 2 samples in 1 batches of 16, ~488.4 target tokens/batch, trg/src length ratio: 0.10 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (376, 129): 2 samples in 1 batches of 16, ~503.9 target tokens/batch, trg/src length ratio: 0.08 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (384, 129): 6 samples in 1 batches of 16, ~520.2 target tokens/batch, trg/src length ratio: 0.08 (+-0.04)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (392, 129): 4 samples in 1 batches of 16, ~522.2 target tokens/batch, trg/src length ratio: 0.08 (+-0.00)
[2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (400, 129): 6 samples in 1 batches of 16, ~551.4 target tokens/batch, trg/src length ratio: 0.08 (+-0.01)
[2023-12-17:11:19:24:INFO:sockeye.data_io:load] Created bucketed parallel data set. Introduced padding: source=7.2% target=85.8%)
[2023-12-17:11:19:24:INFO:sockeye.vocab:vocab_to_json] Vocabulary saved to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/vocab.src.0.json"
[2023-12-17:11:19:24:INFO:sockeye.vocab:vocab_to_json] Vocabulary saved to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/vocab.trg.0.json"
[2023-12-17:11:19:24:INFO:__main__:train] Vocabulary sizes: source=[1232] target=[5976]
[2023-12-17:11:19:24:INFO:__main__:get_num_embed] Source embedding size was not set it will automatically be adjusted to match the Transformer source model size (512).
[2023-12-17:11:19:24:INFO:__main__:get_num_embed] Target embedding size was not set it will automatically be adjusted to match the Transformer target model size (512).
[2023-12-17:11:19:24:INFO:__main__:create_optimizer_config] OptimizerConfig(name='adam', running_on_gpu=True, lr=0.0002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0, momentum=0.0, gradient_clipping_type='none', gradient_clipping_threshold=1.0, update_interval=1)
[2023-12-17:11:19:24:INFO:__main__:create_optimizer_config] Gradient accumulation over 1 batch(es) by 1 worker(s). Effective batch size: 512
[2023-12-17:11:19:24:INFO:sockeye.model:__init__] ModelConfig(config_data=DataConfig(data_statistics=DataStatistics(num_sents=354707, num_discarded=20, num_tokens_source=15626084, num_tokens_target=2020240, num_unks_source=1, num_unks_target=116, max_observed_len_source=417, max_observed_len_target=128, size_vocab_source=1232, size_vocab_target=5976, length_ratio_mean=0.16320710693441579, length_ratio_std=0.1371393774100467, buckets=[(8, 8), (16, 16), (24, 24), (32, 32), (40, 40), (48, 48), (56, 56), (64, 64), (72, 72), (80, 80), (88, 88), (96, 96), (104, 104), (112, 112), (120, 120), (128, 128), (136, 129), (144, 129), (152, 129), (160, 129), (168, 129), (176, 129), (184, 129), (192, 129), (200, 129), (208, 129), (216, 129), (224, 129), (232, 129), (240, 129), (248, 129), (256, 129), (264, 129), (272, 129), (280, 129), (288, 129), (296, 129), (304, 129), (312, 129), (320, 129), (328, 129), (336, 129), (344, 129), (352, 129), (360, 129), (368, 129), (376, 129), (384, 129), (392, 129), (400, 129), (408, 129), (416, 129), (424, 129), (432, 129), (440, 129), (448, 129), (456, 129), (464, 129), (472, 129), (480, 129), (488, 129), (496, 129), (504, 129), (512, 129), (513, 129)], num_sents_per_bucket=[1, 29122, 57389, 123411, 36766, 44876, 21849, 6758, 8500, 2765, 3715, 2686, 722, 752, 417, 491, 402, 347, 249, 196, 297, 376, 262, 246, 213, 269, 331, 317, 350, 356, 368, 451, 454, 497, 483, 519, 572, 500, 550, 608, 561, 537, 481, 568, 516, 468, 554, 512, 486, 455, 130, 2, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], average_len_target_per_bucket=[2.0, 4.08629215026444, 4.545226437122044, 4.607174400985307, 4.87920905184137, 4.982440502718691, 5.27049292873817, 5.803196211897, 6.028941176470578, 6.834719710669081, 7.802691790040373, 15.045793000744581, 8.642659279778403, 8.640957446808516, 10.678657074340526, 9.13441955193482, 9.305970149253735, 8.974063400576362, 10.971887550200805, 13.464285714285717, 13.232323232323239, 11.324468085106396, 13.984732824427489, 16.142276422764233, 16.61032863849766, 16.427509293680295, 16.655589123867063, 19.258675078864364, 21.614285714285707, 21.643258426966298, 20.918478260869566, 22.957871396895783, 23.638766519823815, 24.167002012072434, 25.362318840579718, 25.95375722543352, 26.010489510489492, 27.22000000000001, 26.959999999999987, 27.297697368421066, 28.040998217468793, 30.042830540037233, 29.966735966735982, 30.934859154929573, 30.47868217054262, 30.527777777777782, 31.492779783393505, 32.51171874999999, 32.6358024691358, 34.461538461538474, 32.60769230769231, 5.5, 43.0, None, None, None, None, None, None, None, None, None, None, None, None], length_ratio_stats_per_bucket=[(0.3333333333333333, 0.0), (0.28246393697985434, 0.17868752447804973), (0.21840710265332788, 0.1330505772378312), (0.16560142798704922, 0.09581195473826641), (0.13801367492489092, 0.12429965021659338), (0.11863212215522084, 0.1208393385452983), (0.10151133866588294, 0.11072333780515448), (0.09838819717267734, 0.12557601720946082), (0.09394884997066442, 0.16159177653077658), (0.10402554625981722, 0.26083679437294416), (0.1012342945734544, 0.20099512839826167), (0.17281772320739658, 0.257545103018524), (0.10845391475564008, 0.3239959561352876), (0.09935073708696769, 0.3059573403277105), (0.12635011083619693, 0.3907243857496131), (0.08949294838769961, 0.3144956594612652), (0.07007331783529426, 0.05774973922713284), (0.06411727035132861, 0.04726139664600602), (0.07382609782015778, 0.050521761108029695), (0.08590157138956556, 0.05569960282284095), (0.08062786352616935, 0.058790770283216505), (0.06563222282256796, 0.05450274064413921), (0.07765999502184046, 0.05057167990395854), (0.08555873649959676, 0.05960592960682603), (0.08460026909745419, 0.05238594583690578), (0.08041439956489124, 0.04792844419538253), (0.07859928415542815, 0.05072523580179588), (0.08720935617277471, 0.04382289790728185), (0.0945785744419209, 0.05412799726318098), (0.09147639273741005, 0.04623369327444139), (0.08566937441195915, 0.03729904156233976), (0.09089250053653752, 0.03997864088737986), (0.09074467762227163, 0.03321456251239067), (0.09005889312946631, 0.03317108879820214), (0.09171046510420815, 0.035431771966381115), (0.09113688227781093, 0.03302006652634936), (0.08893255045731784, 0.03625826248767216), (0.09050986731593187, 0.03584124694886162), (0.08733945286928807, 0.02963045027026122), (0.08629641342788141, 0.030504004395265606), (0.08640318897032012, 0.030675309542199148), (0.09036990980396087, 0.03191849333402471), (0.08807433400728253, 0.03290646726223996), (0.08866806731559626, 0.03728726453805084), (0.08543343575292071, 0.02718762541630789), (0.08370713516111211, 0.030215511135668078), (0.08456814843330018, 0.02546164231510412), (0.08555768005742968, 0.026243070552042298), (0.08406679695149158, 0.02438826052491033), (0.08695308402142944, 0.02926245130243095), (0.08093989759976632, 0.032639773078865474), (0.013364278458885503, 0.0011988283372310012), (0.10311750599520383, 0.007770672300249229), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None)]), max_seq_len_source=513, max_seq_len_target=129, num_source_factors=1, num_target_factors=1, eop_id=-1), vocab_source_size=1232, vocab_target_size=5976, config_embed_source=EmbeddingConfig(vocab_size=1232, num_embed=512, dropout=0.0, num_factors=1, factor_configs=None, allow_sparse_grad=False), config_embed_target=EmbeddingConfig(vocab_size=5976, num_embed=512, dropout=0.0, num_factors=1, factor_configs=None, allow_sparse_grad=False), config_encoder=TransformerConfig(model_size=512, attention_heads=8, feed_forward_num_hidden=2048, act_type='relu', num_layers=6, dropout_attention=0.1, dropout_act=0.1, dropout_prepost=0.1, positional_embedding_type='fixed', preprocess_sequence='n', postprocess_sequence='dr', max_seq_len_source=513, max_seq_len_target=129, decoder_type='transformer', block_prepended_cross_attention=False, use_lhuc=False, depth_key_value=512, use_glu=False), config_decoder=TransformerConfig(model_size=512, attention_heads=8, feed_forward_num_hidden=2048, act_type='relu', num_layers=6, dropout_attention=0.1, dropout_act=0.1, dropout_prepost=0.1, positional_embedding_type='fixed', preprocess_sequence='n', postprocess_sequence='dr', max_seq_len_source=513, max_seq_len_target=129, decoder_type='transformer', block_prepended_cross_attention=False, use_lhuc=False, depth_key_value=512, use_glu=False), config_length_task=None, weight_tying_type='none', lhuc=False, dtype='float32', neural_vocab_selection=None, neural_vocab_selection_block_loss=False)
[2023-12-17:11:19:29:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded/model/params.best" to "cuda:0"
[2023-12-17:11:19:29:INFO:sockeye.utils:log_parameters] # of parameters: 51188568 | trainable: 50859864 (99.36%) | shared: 3065688 (5.99%) | fixed: 328704 (0.64%)
[2023-12-17:11:19:29:INFO:sockeye.utils:log_parameters] Trainable parameters: 
['embedding_source.embedding [(1232, 512), float32]',
 'embedding_target.embedding [(5976, 512), float32]',
 'encoder.layers.0.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.0.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.0.self_attention.ff_out [(512, 512), float32]',
 'encoder.layers.0.self_attention.ff_in [(1536, 512), float32]',
 'encoder.layers.0.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.0.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.0.ff.ff1 [(2048, 512), float32]',
 'encoder.layers.0.ff.ff1 [(2048,), float32]',
 'encoder.layers.0.ff.ff2 [(512, 2048), float32]',
 'encoder.layers.0.ff.ff2 [(512,), float32]',
 'encoder.layers.1.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.1.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.1.self_attention.ff_out [(512, 512), float32]',
 'encoder.layers.1.self_attention.ff_in [(1536, 512), float32]',
 'encoder.layers.1.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.1.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.1.ff.ff1 [(2048, 512), float32]',
 'encoder.layers.1.ff.ff1 [(2048,), float32]',
 'encoder.layers.1.ff.ff2 [(512, 2048), float32]',
 'encoder.layers.1.ff.ff2 [(512,), float32]',
 'encoder.layers.2.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.2.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.2.self_attention.ff_out [(512, 512), float32]',
 'encoder.layers.2.self_attention.ff_in [(1536, 512), float32]',
 'encoder.layers.2.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.2.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.2.ff.ff1 [(2048, 512), float32]',
 'encoder.layers.2.ff.ff1 [(2048,), float32]',
 'encoder.layers.2.ff.ff2 [(512, 2048), float32]',
 'encoder.layers.2.ff.ff2 [(512,), float32]',
 'encoder.layers.3.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.3.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.3.self_attention.ff_out [(512, 512), float32]',
 'encoder.layers.3.self_attention.ff_in [(1536, 512), float32]',
 'encoder.layers.3.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.3.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.3.ff.ff1 [(2048, 512), float32]',
 'encoder.layers.3.ff.ff1 [(2048,), float32]',
 'encoder.layers.3.ff.ff2 [(512, 2048), float32]',
 'encoder.layers.3.ff.ff2 [(512,), float32]',
 'encoder.layers.4.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.4.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.4.self_attention.ff_out [(512, 512), float32]',
 'encoder.layers.4.self_attention.ff_in [(1536, 512), float32]',
 'encoder.layers.4.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.4.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.4.ff.ff1 [(2048, 512), float32]',
 'encoder.layers.4.ff.ff1 [(2048,), float32]',
 'encoder.layers.4.ff.ff2 [(512, 2048), float32]',
 'encoder.layers.4.ff.ff2 [(512,), float32]',
 'encoder.layers.5.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.5.pre_self_attention.layer_norm [(512,), float32]',
 'encoder.layers.5.self_attention.ff_out [(512, 512), float32]',
 'encoder.layers.5.self_attention.ff_in [(1536, 512), float32]',
 'encoder.layers.5.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.5.pre_ff.layer_norm [(512,), float32]',
 'encoder.layers.5.ff.ff1 [(2048, 512), float32]',
 'encoder.layers.5.ff.ff1 [(2048,), float32]',
 'encoder.layers.5.ff.ff2 [(512, 2048), float32]',
 'encoder.layers.5.ff.ff2 [(512,), float32]',
 'encoder.final_process.layer_norm [(512,), float32]',
 'encoder.final_process.layer_norm [(512,), float32]',
 'decoder.layers.0.autoregr_layer.ff_out [(512, 512), float32]',
 'decoder.layers.0.autoregr_layer.ff_in [(1536, 512), float32]',
 'decoder.layers.0.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.0.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.0.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.0.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.0.enc_attention.ff_out [(512, 512), float32]',
 'decoder.layers.0.enc_attention.ff_q [(512, 512), float32]',
 'decoder.layers.0.enc_attention.ff_kv [(1024, 512), float32]',
 'decoder.layers.0.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.0.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.0.ff.ff1 [(2048, 512), float32]',
 'decoder.layers.0.ff.ff1 [(2048,), float32]',
 'decoder.layers.0.ff.ff2 [(512, 2048), float32]',
 'decoder.layers.0.ff.ff2 [(512,), float32]',
 'decoder.layers.1.autoregr_layer.ff_out [(512, 512), float32]',
 'decoder.layers.1.autoregr_layer.ff_in [(1536, 512), float32]',
 'decoder.layers.1.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.1.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.1.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.1.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.1.enc_attention.ff_out [(512, 512), float32]',
 'decoder.layers.1.enc_attention.ff_q [(512, 512), float32]',
 'decoder.layers.1.enc_attention.ff_kv [(1024, 512), float32]',
 'decoder.layers.1.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.1.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.1.ff.ff1 [(2048, 512), float32]',
 'decoder.layers.1.ff.ff1 [(2048,), float32]',
 'decoder.layers.1.ff.ff2 [(512, 2048), float32]',
 'decoder.layers.1.ff.ff2 [(512,), float32]',
 'decoder.layers.2.autoregr_layer.ff_out [(512, 512), float32]',
 'decoder.layers.2.autoregr_layer.ff_in [(1536, 512), float32]',
 'decoder.layers.2.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.2.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.2.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.2.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.2.enc_attention.ff_out [(512, 512), float32]',
 'decoder.layers.2.enc_attention.ff_q [(512, 512), float32]',
 'decoder.layers.2.enc_attention.ff_kv [(1024, 512), float32]',
 'decoder.layers.2.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.2.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.2.ff.ff1 [(2048, 512), float32]',
 'decoder.layers.2.ff.ff1 [(2048,), float32]',
 'decoder.layers.2.ff.ff2 [(512, 2048), float32]',
 'decoder.layers.2.ff.ff2 [(512,), float32]',
 'decoder.layers.3.autoregr_layer.ff_out [(512, 512), float32]',
 'decoder.layers.3.autoregr_layer.ff_in [(1536, 512), float32]',
 'decoder.layers.3.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.3.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.3.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.3.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.3.enc_attention.ff_out [(512, 512), float32]',
 'decoder.layers.3.enc_attention.ff_q [(512, 512), float32]',
 'decoder.layers.3.enc_attention.ff_kv [(1024, 512), float32]',
 'decoder.layers.3.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.3.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.3.ff.ff1 [(2048, 512), float32]',
 'decoder.layers.3.ff.ff1 [(2048,), float32]',
 'decoder.layers.3.ff.ff2 [(512, 2048), float32]',
 'decoder.layers.3.ff.ff2 [(512,), float32]',
 'decoder.layers.4.autoregr_layer.ff_out [(512, 512), float32]',
 'decoder.layers.4.autoregr_layer.ff_in [(1536, 512), float32]',
 'decoder.layers.4.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.4.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.4.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.4.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.4.enc_attention.ff_out [(512, 512), float32]',
 'decoder.layers.4.enc_attention.ff_q [(512, 512), float32]',
 'decoder.layers.4.enc_attention.ff_kv [(1024, 512), float32]',
 'decoder.layers.4.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.4.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.4.ff.ff1 [(2048, 512), float32]',
 'decoder.layers.4.ff.ff1 [(2048,), float32]',
 'decoder.layers.4.ff.ff2 [(512, 2048), float32]',
 'decoder.layers.4.ff.ff2 [(512,), float32]',
 'decoder.layers.5.autoregr_layer.ff_out [(512, 512), float32]',
 'decoder.layers.5.autoregr_layer.ff_in [(1536, 512), float32]',
 'decoder.layers.5.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.5.pre_autoregr_layer.layer_norm [(512,), float32]',
 'decoder.layers.5.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.5.pre_enc_attention.layer_norm [(512,), float32]',
 'decoder.layers.5.enc_attention.ff_out [(512, 512), float32]',
 'decoder.layers.5.enc_attention.ff_q [(512, 512), float32]',
 'decoder.layers.5.enc_attention.ff_kv [(1024, 512), float32]',
 'decoder.layers.5.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.5.pre_ff.layer_norm [(512,), float32]',
 'decoder.layers.5.ff.ff1 [(2048, 512), float32]',
 'decoder.layers.5.ff.ff1 [(2048,), float32]',
 'decoder.layers.5.ff.ff2 [(512, 2048), float32]',
 'decoder.layers.5.ff.ff2 [(512,), float32]',
 'decoder.final_process.layer_norm [(512,), float32]',
 'decoder.final_process.layer_norm [(512,), float32]',
 'output_layer [(5976, 512), float32]',
 'output_layer [(5976,), float32]',
 'output_layer_module_cached [(5976, 512), float32]',
 'output_layer_module_cached [(5976,), float32]',
 'output_layer_script_cached [(5976, 512), float32]',
 'output_layer_script_cached [(5976,), float32]']
[2023-12-17:11:19:29:INFO:sockeye.utils:log_parameters] Shared parameters: 
['output_layer.weight = output_layer_module_cached.weight = output_layer_script_cached.weight',
 'output_layer.bias = output_layer_module_cached.bias = output_layer_script_cached.bias']
[2023-12-17:11:19:29:INFO:sockeye.utils:log_parameters] Fixed parameters:
['encoder.pos_embedding [(513, 512), float32]',
 'decoder.pos_embedding [(129, 512), float32]']
[2023-12-17:11:19:29:INFO:sockeye.loss:__init__] Loss: cross-entropy | weight=1.00 | metric: perplexity (ppl) | output_name: 'logits' | label_name: 'target_label'
[2023-12-17:11:19:29:WARNING:sockeye.optimizers:get_optimizer] Cannot import NVIDIA Apex optimizers (FusedAdam, FusedSGD). Consider installing Apex for faster GPU training: https://github.com/NVIDIA/apex
[2023-12-17:11:19:29:INFO:sockeye.lr_scheduler:__init__] Will reduce the learning rate by a factor of 0.90 whenever the validation score doesn't improve 8 times.
[2023-12-17:11:19:29:INFO:__main__:train] Tracing SockeyeModel on a validation batch
[2023-12-17:11:19:33:INFO:sockeye.training:__init__] tensorboard not found. Consider 'pip install tensorboard' to log events to Tensorboard.
[2023-12-17:11:19:33:INFO:sockeye.inference:__init__] Translator (1 model(s) beam_size=5 algorithm=BeamSearch, beam_search_stop=all max_input_length=512 nbest_size=1 ensemble_mode=None max_batch_size=16 dtype=torch.float32 skip_nvs=False nvs_thresh=0.5)
[2023-12-17:11:19:33:INFO:sockeye.checkpoint_decoder:__init__] Created CheckpointDecoder(max_input_len=-1, beam_size=5, num_sentences=500)
[2023-12-17:11:20:03:INFO:sockeye.training:fit] Early stopping by optimizing 'chrf'
[2023-12-17:11:20:03:INFO:sockeye.model:save_config] Saved model config to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/config"
[2023-12-17:11:20:03:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00000"
[2023-12-17:11:20:03:INFO:sockeye.training:fit] Training started.
[2023-12-17:11:21:06:INFO:sockeye.training:__call__] E=0 B=50	s/sec=74.49 tok/sec=3434.09 u/sec=0.81	ppl=18.655668 
[2023-12-17:11:21:08:INFO:sockeye.training:__call__] E=0 B=100	s/sec=2336.30 tok/sec=104114.57 u/sec=24.22	ppl=18.608390 
[2023-12-17:11:21:10:INFO:sockeye.training:__call__] E=0 B=150	s/sec=2143.73 tok/sec=105962.53 u/sec=24.67	ppl=18.206416 
[2023-12-17:11:21:12:INFO:sockeye.training:__call__] E=0 B=200	s/sec=2353.40 tok/sec=106096.56 u/sec=25.19	ppl=18.171534 
[2023-12-17:11:21:14:INFO:sockeye.training:__call__] E=0 B=250	s/sec=2212.65 tok/sec=108159.94 u/sec=24.61	ppl=18.142329 
[2023-12-17:11:21:16:INFO:sockeye.training:__call__] E=0 B=300	s/sec=2669.99 tok/sec=102130.10 u/sec=25.87	ppl=18.145817 
[2023-12-17:11:21:18:INFO:sockeye.training:__call__] E=0 B=350	s/sec=1913.96 tok/sec=113237.37 u/sec=24.51	ppl=18.044639 
[2023-12-17:11:21:20:INFO:sockeye.training:__call__] E=0 B=400	s/sec=2043.56 tok/sec=109010.56 u/sec=24.37	ppl=17.931931 
[2023-12-17:11:21:22:INFO:sockeye.training:__call__] E=0 B=450	s/sec=2223.97 tok/sec=110042.76 u/sec=24.73	ppl=17.854408 
[2023-12-17:11:21:24:INFO:sockeye.training:__call__] E=0 B=500	s/sec=2338.26 tok/sec=106896.46 u/sec=25.59	ppl=17.848924 
[2023-12-17:11:21:26:INFO:sockeye.training:__call__] E=0 B=550	s/sec=2256.38 tok/sec=107088.11 u/sec=25.05	ppl=17.821833 
[2023-12-17:11:21:28:INFO:sockeye.training:__call__] E=0 B=600	s/sec=2287.93 tok/sec=105791.94 u/sec=24.78	ppl=17.818209 
[2023-12-17:11:21:30:INFO:sockeye.training:__call__] E=0 B=650	s/sec=2003.44 tok/sec=108956.76 u/sec=23.85	ppl=17.794558 
[2023-12-17:11:21:32:INFO:sockeye.training:__call__] E=0 B=700	s/sec=1977.20 tok/sec=109670.89 u/sec=24.91	ppl=17.729489 
[2023-12-17:11:21:34:INFO:sockeye.training:__call__] E=0 B=750	s/sec=1917.73 tok/sec=112075.36 u/sec=24.07	ppl=17.731481 
[2023-12-17:11:21:36:INFO:sockeye.training:__call__] E=0 B=800	s/sec=2322.90 tok/sec=107205.81 u/sec=25.03	ppl=17.737991 
[2023-12-17:11:21:38:INFO:sockeye.training:__call__] E=0 B=850	s/sec=2243.45 tok/sec=108070.36 u/sec=25.04	ppl=17.691493 
[2023-12-17:11:21:40:INFO:sockeye.training:__call__] E=0 B=900	s/sec=2141.74 tok/sec=108902.87 u/sec=23.95	ppl=17.670818 
[2023-12-17:11:21:42:INFO:sockeye.training:__call__] E=0 B=950	s/sec=2143.04 tok/sec=105702.48 u/sec=24.62	ppl=17.615685 
[2023-12-17:11:21:44:INFO:sockeye.training:__call__] E=0 B=1000	s/sec=2139.96 tok/sec=108931.79 u/sec=24.77	ppl=17.619012 
[2023-12-17:11:21:46:INFO:sockeye.training:__call__] E=0 B=1050	s/sec=2140.44 tok/sec=107687.76 u/sec=24.77	ppl=17.588894 
[2023-12-17:11:21:48:INFO:sockeye.training:__call__] E=0 B=1100	s/sec=2443.49 tok/sec=105328.87 u/sec=25.67	ppl=17.617007 
[2023-12-17:11:21:51:INFO:sockeye.training:__call__] E=0 B=1150	s/sec=1705.10 tok/sec=102555.01 u/sec=19.45	ppl=17.577034 
[2023-12-17:11:21:53:INFO:sockeye.training:__call__] E=0 B=1200	s/sec=2241.08 tok/sec=106864.79 u/sec=25.10	ppl=17.573240 
[2023-12-17:11:21:55:INFO:sockeye.training:__call__] E=0 B=1250	s/sec=2339.28 tok/sec=102717.32 u/sec=25.34	ppl=17.568516 
[2023-12-17:11:21:57:INFO:sockeye.training:__call__] E=0 B=1300	s/sec=2094.58 tok/sec=110167.94 u/sec=24.11	ppl=17.560434 
[2023-12-17:11:21:59:INFO:sockeye.training:__call__] E=0 B=1350	s/sec=2302.40 tok/sec=106618.90 u/sec=25.16	ppl=17.541674 
[2023-12-17:11:22:01:INFO:sockeye.training:__call__] E=0 B=1400	s/sec=2500.07 tok/sec=99702.29 u/sec=26.09	ppl=17.561718 
[2023-12-17:11:22:03:INFO:sockeye.training:__call__] E=0 B=1450	s/sec=2197.94 tok/sec=105695.57 u/sec=25.30	ppl=17.544423 
[2023-12-17:11:22:04:INFO:sockeye.training:__call__] E=0 B=1500	s/sec=2291.61 tok/sec=105721.71 u/sec=25.30	ppl=17.518426 
[2023-12-17:11:22:06:INFO:sockeye.training:__call__] E=0 B=1550	s/sec=2311.30 tok/sec=106807.95 u/sec=25.43	ppl=17.509666 
[2023-12-17:11:22:08:INFO:sockeye.training:__call__] E=0 B=1600	s/sec=2014.89 tok/sec=111298.68 u/sec=24.99	ppl=17.483595 
[2023-12-17:11:22:10:INFO:sockeye.training:__call__] E=0 B=1650	s/sec=2036.85 tok/sec=107306.52 u/sec=25.26	ppl=17.450279 
[2023-12-17:11:22:12:INFO:sockeye.training:__call__] E=0 B=1700	s/sec=2217.83 tok/sec=107323.71 u/sec=25.11	ppl=17.437040 
[2023-12-17:11:22:14:INFO:sockeye.training:__call__] E=0 B=1750	s/sec=2381.45 tok/sec=105723.47 u/sec=25.31	ppl=17.438608 
[2023-12-17:11:22:16:INFO:sockeye.training:__call__] E=0 B=1800	s/sec=2232.63 tok/sec=108222.69 u/sec=25.01	ppl=17.423068 
[2023-12-17:11:22:18:INFO:sockeye.training:__call__] E=0 B=1850	s/sec=2419.68 tok/sec=104562.83 u/sec=25.63	ppl=17.400424 
[2023-12-17:11:22:20:INFO:sockeye.training:__call__] E=0 B=1900	s/sec=2112.82 tok/sec=108239.43 u/sec=24.68	ppl=17.392888 
[2023-12-17:11:22:22:INFO:sockeye.training:__call__] E=0 B=1950	s/sec=2145.62 tok/sec=108062.43 u/sec=24.93	ppl=17.394044 
[2023-12-17:11:22:24:INFO:sockeye.training:__call__] E=0 B=2000	s/sec=2343.37 tok/sec=101339.48 u/sec=25.83	ppl=17.364784 
[2023-12-17:11:22:26:INFO:sockeye.training:__call__] E=0 B=2050	s/sec=2532.08 tok/sec=103286.03 u/sec=25.65	ppl=17.370202 
[2023-12-17:11:22:28:INFO:sockeye.training:__call__] E=0 B=2100	s/sec=2451.83 tok/sec=104512.98 u/sec=25.67	ppl=17.363843 
[2023-12-17:11:22:30:INFO:sockeye.training:__call__] E=0 B=2150	s/sec=2316.15 tok/sec=106574.82 u/sec=24.79	ppl=17.346560 
[2023-12-17:11:22:32:INFO:sockeye.training:__call__] E=0 B=2200	s/sec=2413.29 tok/sec=103641.06 u/sec=25.48	ppl=17.336061 
[2023-12-17:11:22:34:INFO:sockeye.training:__call__] E=0 B=2250	s/sec=2295.75 tok/sec=105572.35 u/sec=25.26	ppl=17.321225 
[2023-12-17:11:22:36:INFO:sockeye.training:__call__] E=0 B=2300	s/sec=2298.07 tok/sec=106597.76 u/sec=25.42	ppl=17.332496 
[2023-12-17:11:22:38:INFO:sockeye.training:__call__] E=0 B=2350	s/sec=2143.55 tok/sec=108318.89 u/sec=24.95	ppl=17.326014 
[2023-12-17:11:22:40:INFO:sockeye.training:__call__] E=0 B=2400	s/sec=2281.87 tok/sec=103521.53 u/sec=25.65	ppl=17.303162 
[2023-12-17:11:22:42:INFO:sockeye.training:__call__] E=0 B=2450	s/sec=2369.40 tok/sec=107841.93 u/sec=25.14	ppl=17.311880 
[2023-12-17:11:22:44:INFO:sockeye.training:__call__] E=0 B=2500	s/sec=2322.30 tok/sec=105678.93 u/sec=25.42	ppl=17.309172 
[2023-12-17:11:22:46:INFO:sockeye.training:__call__] E=0 B=2550	s/sec=2241.69 tok/sec=108404.40 u/sec=25.11	ppl=17.305746 
[2023-12-17:11:22:48:INFO:sockeye.training:__call__] E=0 B=2600	s/sec=2369.07 tok/sec=105001.36 u/sec=25.89	ppl=17.295458 
[2023-12-17:11:22:50:INFO:sockeye.training:__call__] E=0 B=2650	s/sec=2157.39 tok/sec=109138.61 u/sec=24.79	ppl=17.283534 
[2023-12-17:11:22:52:INFO:sockeye.training:__call__] E=0 B=2700	s/sec=2333.57 tok/sec=107230.36 u/sec=25.41	ppl=17.280709 
[2023-12-17:11:22:54:INFO:sockeye.training:__call__] E=0 B=2750	s/sec=2552.08 tok/sec=101848.12 u/sec=26.23	ppl=17.274363 
[2023-12-17:11:22:56:INFO:sockeye.training:__call__] E=0 B=2800	s/sec=1847.99 tok/sec=114121.98 u/sec=24.01	ppl=17.267134 
[2023-12-17:11:22:58:INFO:sockeye.training:__call__] E=0 B=2850	s/sec=2237.11 tok/sec=107030.57 u/sec=24.92	ppl=17.267797 
[2023-12-17:11:23:00:INFO:sockeye.training:__call__] E=0 B=2900	s/sec=2101.49 tok/sec=106788.73 u/sec=24.46	ppl=17.250172 
[2023-12-17:11:23:02:INFO:sockeye.training:__call__] E=0 B=2950	s/sec=2104.02 tok/sec=109694.65 u/sec=24.81	ppl=17.241270 
[2023-12-17:11:23:04:INFO:sockeye.training:__call__] E=0 B=3000	s/sec=2358.34 tok/sec=103750.13 u/sec=26.00	ppl=17.233959 
[2023-12-17:11:23:06:INFO:sockeye.training:__call__] E=0 B=3050	s/sec=2068.26 tok/sec=109135.07 u/sec=24.53	ppl=17.230318 
[2023-12-17:11:23:08:INFO:sockeye.training:__call__] E=0 B=3100	s/sec=2576.00 tok/sec=102635.79 u/sec=26.31	ppl=17.234948 
[2023-12-17:11:23:10:INFO:sockeye.training:__call__] E=0 B=3150	s/sec=1994.04 tok/sec=110714.60 u/sec=24.78	ppl=17.233230 
[2023-12-17:11:23:12:INFO:sockeye.training:__call__] E=0 B=3200	s/sec=1958.45 tok/sec=108967.01 u/sec=25.08	ppl=17.212959 
[2023-12-17:11:23:14:INFO:sockeye.training:__call__] E=0 B=3250	s/sec=2463.45 tok/sec=104839.28 u/sec=25.12	ppl=17.213881 
[2023-12-17:11:23:16:INFO:sockeye.training:__call__] E=0 B=3300	s/sec=2195.44 tok/sec=108452.42 u/sec=24.99	ppl=17.212524 
[2023-12-17:11:23:18:INFO:sockeye.training:__call__] E=0 B=3350	s/sec=2145.07 tok/sec=108704.53 u/sec=24.78	ppl=17.206518 
[2023-12-17:11:23:20:INFO:sockeye.training:__call__] E=0 B=3400	s/sec=2041.84 tok/sec=113002.81 u/sec=24.35	ppl=17.200257 
[2023-12-17:11:23:22:INFO:sockeye.training:__call__] E=0 B=3450	s/sec=2266.35 tok/sec=107219.52 u/sec=25.48	ppl=17.189116 
[2023-12-17:11:23:24:INFO:sockeye.training:__call__] E=0 B=3500	s/sec=2234.35 tok/sec=107509.51 u/sec=25.44	ppl=17.176347 
[2023-12-17:11:23:26:INFO:sockeye.training:__call__] E=0 B=3550	s/sec=2147.71 tok/sec=105614.64 u/sec=25.62	ppl=17.161121 
[2023-12-17:11:23:28:INFO:sockeye.training:__call__] E=0 B=3600	s/sec=2065.38 tok/sec=108923.99 u/sec=25.07	ppl=17.152146 
[2023-12-17:11:23:30:INFO:sockeye.training:__call__] E=0 B=3650	s/sec=2364.99 tok/sec=107029.80 u/sec=25.22	ppl=17.141910 
[2023-12-17:11:23:32:INFO:sockeye.training:__call__] E=0 B=3700	s/sec=2103.91 tok/sec=108482.59 u/sec=25.24	ppl=17.141180 
[2023-12-17:11:23:34:INFO:sockeye.training:__call__] E=0 B=3750	s/sec=2154.56 tok/sec=107361.53 u/sec=25.46	ppl=17.122229 
[2023-12-17:11:23:36:INFO:sockeye.training:__call__] E=0 B=3800	s/sec=2253.65 tok/sec=107259.04 u/sec=25.56	ppl=17.113981 
[2023-12-17:11:23:38:INFO:sockeye.training:__call__] E=0 B=3850	s/sec=2332.09 tok/sec=106310.37 u/sec=25.57	ppl=17.113770 
[2023-12-17:11:23:40:INFO:sockeye.training:__call__] E=0 B=3900	s/sec=2314.67 tok/sec=108216.83 u/sec=24.52	ppl=17.106544 
[2023-12-17:11:23:42:INFO:sockeye.training:__call__] E=0 B=3950	s/sec=2416.28 tok/sec=104743.67 u/sec=25.60	ppl=17.104703 
[2023-12-17:11:23:44:INFO:sockeye.training:__call__] E=1 B=4000	s/sec=2193.01 tok/sec=100832.62 u/sec=23.88	ppl=17.100023 
[2023-12-17:11:23:44:INFO:sockeye.training:_create_checkpoint] Checkpoint [1]	Updates=4000 Epoch=1 Samples=356496 Time-cost=220.406 Updates/sec=18.148
[2023-12-17:11:23:44:INFO:sockeye.training:_create_checkpoint] Checkpoint [1]	Train-ppl=17.100023
[2023-12-17:11:25:05:INFO:sockeye.training:_evaluate] Checkpoint [1]	Validation-ppl=22.523855	Validation-bleu=0.127874	Validation-chrf=28.768187	Validation-rouge1=0.157579	Validation-rouge2=0.074995	Validation-rougel=0.151230	Validation-length-ratio-mse=1.283504	Validation-ter=97.234833	Validation-avg-sec-per-sent=0.160033	Validation-decode-walltime=80.016319
[2023-12-17:11:25:05:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 28.768187 (delta=28.768187).
[2023-12-17:11:25:05:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [1]	Learning-rate=0.000200
[2023-12-17:11:25:05:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:25:05:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:25:06:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00001"
[2023-12-17:11:25:06:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00001'
[2023-12-17:11:25:06:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:11:25:06:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl'
[2023-12-17:11:25:08:INFO:sockeye.training:__call__] E=1 B=4050	s/sec=55.36 tok/sec=2426.19 u/sec=0.59	ppl=12.546023 
[2023-12-17:11:25:10:INFO:sockeye.training:__call__] E=1 B=4100	s/sec=2266.37 tok/sec=107137.49 u/sec=24.76	ppl=12.526253 
[2023-12-17:11:25:12:INFO:sockeye.training:__call__] E=1 B=4150	s/sec=2506.03 tok/sec=104094.26 u/sec=25.51	ppl=12.690548 
[2023-12-17:11:25:14:INFO:sockeye.training:__call__] E=1 B=4200	s/sec=2000.83 tok/sec=109130.50 u/sec=25.31	ppl=12.582174 
[2023-12-17:11:25:16:INFO:sockeye.training:__call__] E=1 B=4250	s/sec=2383.59 tok/sec=105350.53 u/sec=25.47	ppl=12.589332 
[2023-12-17:11:25:18:INFO:sockeye.training:__call__] E=1 B=4300	s/sec=1981.98 tok/sec=111549.48 u/sec=24.24	ppl=12.600232 
[2023-12-17:11:25:20:INFO:sockeye.training:__call__] E=1 B=4350	s/sec=2434.26 tok/sec=105757.29 u/sec=24.82	ppl=12.679638 
[2023-12-17:11:25:22:INFO:sockeye.training:__call__] E=1 B=4400	s/sec=2090.83 tok/sec=107477.75 u/sec=25.28	ppl=12.692391 
[2023-12-17:11:25:24:INFO:sockeye.training:__call__] E=1 B=4450	s/sec=2255.01 tok/sec=106352.03 u/sec=25.44	ppl=12.715244 
[2023-12-17:11:25:26:INFO:sockeye.training:__call__] E=1 B=4500	s/sec=2468.43 tok/sec=104014.05 u/sec=25.29	ppl=12.760256 
[2023-12-17:11:25:28:INFO:sockeye.training:__call__] E=1 B=4550	s/sec=2240.17 tok/sec=108569.51 u/sec=24.65	ppl=12.793539 
[2023-12-17:11:25:30:INFO:sockeye.training:__call__] E=1 B=4600	s/sec=2031.87 tok/sec=107564.89 u/sec=25.05	ppl=12.783694 
[2023-12-17:11:25:32:INFO:sockeye.training:__call__] E=1 B=4650	s/sec=2187.24 tok/sec=106870.18 u/sec=24.11	ppl=12.774624 
[2023-12-17:11:25:34:INFO:sockeye.training:__call__] E=1 B=4700	s/sec=2399.69 tok/sec=105406.31 u/sec=25.55	ppl=12.777491 
[2023-12-17:11:25:36:INFO:sockeye.training:__call__] E=1 B=4750	s/sec=2397.24 tok/sec=104377.03 u/sec=25.31	ppl=12.822910 
[2023-12-17:11:25:38:INFO:sockeye.training:__call__] E=1 B=4800	s/sec=2161.12 tok/sec=110515.69 u/sec=24.87	ppl=12.831135 
[2023-12-17:11:25:40:INFO:sockeye.training:__call__] E=1 B=4850	s/sec=2090.30 tok/sec=110944.35 u/sec=24.74	ppl=12.850528 
[2023-12-17:11:25:42:INFO:sockeye.training:__call__] E=1 B=4900	s/sec=2312.34 tok/sec=107075.96 u/sec=24.87	ppl=12.868678 
[2023-12-17:11:25:44:INFO:sockeye.training:__call__] E=1 B=4950	s/sec=2498.47 tok/sec=103871.53 u/sec=25.39	ppl=12.926488 
[2023-12-17:11:25:46:INFO:sockeye.training:__call__] E=1 B=5000	s/sec=2368.04 tok/sec=108084.60 u/sec=24.92	ppl=12.973971 
[2023-12-17:11:25:48:INFO:sockeye.training:__call__] E=1 B=5050	s/sec=1941.29 tok/sec=106541.83 u/sec=23.88	ppl=13.002768 
[2023-12-17:11:25:50:INFO:sockeye.training:__call__] E=1 B=5100	s/sec=2188.98 tok/sec=106450.36 u/sec=25.06	ppl=13.029866 
[2023-12-17:11:25:52:INFO:sockeye.training:__call__] E=1 B=5150	s/sec=2350.78 tok/sec=104982.85 u/sec=24.94	ppl=13.058025 
[2023-12-17:11:25:54:INFO:sockeye.training:__call__] E=1 B=5200	s/sec=2192.34 tok/sec=107880.69 u/sec=25.23	ppl=13.076669 
[2023-12-17:11:25:56:INFO:sockeye.training:__call__] E=1 B=5250	s/sec=2252.10 tok/sec=109027.27 u/sec=24.96	ppl=13.099099 
[2023-12-17:11:25:58:INFO:sockeye.training:__call__] E=1 B=5300	s/sec=2126.77 tok/sec=108129.42 u/sec=24.99	ppl=13.110970 
[2023-12-17:11:26:00:INFO:sockeye.training:__call__] E=1 B=5350	s/sec=2069.00 tok/sec=111410.09 u/sec=24.68	ppl=13.117524 
[2023-12-17:11:26:02:INFO:sockeye.training:__call__] E=1 B=5400	s/sec=2106.33 tok/sec=116710.60 u/sec=23.14	ppl=13.126858 
[2023-12-17:11:26:04:INFO:sockeye.training:__call__] E=1 B=5450	s/sec=2435.14 tok/sec=103689.82 u/sec=25.84	ppl=13.149983 
[2023-12-17:11:26:06:INFO:sockeye.training:__call__] E=1 B=5500	s/sec=2064.90 tok/sec=108371.03 u/sec=25.11	ppl=13.160420 
[2023-12-17:11:26:08:INFO:sockeye.training:__call__] E=1 B=5550	s/sec=2182.28 tok/sec=108247.18 u/sec=24.62	ppl=13.178994 
[2023-12-17:11:26:10:INFO:sockeye.training:__call__] E=1 B=5600	s/sec=2136.41 tok/sec=108412.98 u/sec=24.64	ppl=13.193953 
[2023-12-17:11:26:12:INFO:sockeye.training:__call__] E=1 B=5650	s/sec=2160.95 tok/sec=106333.04 u/sec=25.15	ppl=13.193265 
[2023-12-17:11:26:14:INFO:sockeye.training:__call__] E=1 B=5700	s/sec=2227.10 tok/sec=98617.36 u/sec=24.42	ppl=13.201182 
[2023-12-17:11:26:16:INFO:sockeye.training:__call__] E=1 B=5750	s/sec=2298.46 tok/sec=104373.14 u/sec=25.84	ppl=13.201866 
[2023-12-17:11:26:18:INFO:sockeye.training:__call__] E=1 B=5800	s/sec=2031.43 tok/sec=107786.04 u/sec=25.09	ppl=13.205235 
[2023-12-17:11:26:20:INFO:sockeye.training:__call__] E=1 B=5850	s/sec=2468.43 tok/sec=101243.65 u/sec=26.46	ppl=13.217665 
[2023-12-17:11:26:22:INFO:sockeye.training:__call__] E=1 B=5900	s/sec=2571.65 tok/sec=104561.32 u/sec=25.47	ppl=13.231522 
[2023-12-17:11:26:24:INFO:sockeye.training:__call__] E=1 B=5950	s/sec=2325.27 tok/sec=106760.12 u/sec=25.27	ppl=13.241720 
[2023-12-17:11:26:26:INFO:sockeye.training:__call__] E=1 B=6000	s/sec=1966.04 tok/sec=109630.20 u/sec=24.38	ppl=13.256885 
[2023-12-17:11:26:28:INFO:sockeye.training:__call__] E=1 B=6050	s/sec=2528.83 tok/sec=103801.99 u/sec=25.87	ppl=13.280794 
[2023-12-17:11:26:30:INFO:sockeye.training:__call__] E=1 B=6100	s/sec=2206.18 tok/sec=105671.38 u/sec=25.16	ppl=13.286274 
[2023-12-17:11:26:32:INFO:sockeye.training:__call__] E=1 B=6150	s/sec=2417.30 tok/sec=105099.89 u/sec=25.26	ppl=13.304885 
[2023-12-17:11:26:34:INFO:sockeye.training:__call__] E=1 B=6200	s/sec=2152.95 tok/sec=108124.27 u/sec=25.25	ppl=13.316943 
[2023-12-17:11:26:36:INFO:sockeye.training:__call__] E=1 B=6250	s/sec=2499.69 tok/sec=104216.61 u/sec=25.61	ppl=13.333610 
[2023-12-17:11:26:38:INFO:sockeye.training:__call__] E=1 B=6300	s/sec=2186.63 tok/sec=109834.86 u/sec=24.58	ppl=13.349518 
[2023-12-17:11:26:40:INFO:sockeye.training:__call__] E=1 B=6350	s/sec=2240.49 tok/sec=108143.90 u/sec=24.70	ppl=13.362637 
[2023-12-17:11:26:42:INFO:sockeye.training:__call__] E=1 B=6400	s/sec=2118.71 tok/sec=108649.10 u/sec=24.80	ppl=13.372965 
[2023-12-17:11:26:44:INFO:sockeye.training:__call__] E=1 B=6450	s/sec=2323.65 tok/sec=105552.31 u/sec=25.00	ppl=13.377158 
[2023-12-17:11:26:46:INFO:sockeye.training:__call__] E=1 B=6500	s/sec=1970.17 tok/sec=109496.44 u/sec=25.03	ppl=13.375593 
[2023-12-17:11:26:48:INFO:sockeye.training:__call__] E=1 B=6550	s/sec=2559.75 tok/sec=103018.36 u/sec=25.85	ppl=13.393016 
[2023-12-17:11:26:50:INFO:sockeye.training:__call__] E=1 B=6600	s/sec=2258.82 tok/sec=104926.05 u/sec=25.30	ppl=13.406176 
[2023-12-17:11:26:52:INFO:sockeye.training:__call__] E=1 B=6650	s/sec=2282.19 tok/sec=104792.43 u/sec=25.61	ppl=13.415947 
[2023-12-17:11:26:54:INFO:sockeye.training:__call__] E=1 B=6700	s/sec=2275.23 tok/sec=103977.39 u/sec=25.71	ppl=13.436465 
[2023-12-17:11:26:56:INFO:sockeye.training:__call__] E=1 B=6750	s/sec=2274.26 tok/sec=106764.39 u/sec=25.34	ppl=13.445209 
[2023-12-17:11:26:58:INFO:sockeye.training:__call__] E=1 B=6800	s/sec=2261.88 tok/sec=107134.01 u/sec=24.93	ppl=13.447806 
[2023-12-17:11:27:00:INFO:sockeye.training:__call__] E=1 B=6850	s/sec=2086.05 tok/sec=106348.51 u/sec=25.17	ppl=13.455241 
[2023-12-17:11:27:02:INFO:sockeye.training:__call__] E=1 B=6900	s/sec=2246.42 tok/sec=109863.91 u/sec=24.85	ppl=13.467257 
[2023-12-17:11:27:04:INFO:sockeye.training:__call__] E=1 B=6950	s/sec=2436.83 tok/sec=105674.67 u/sec=25.43	ppl=13.479890 
[2023-12-17:11:27:06:INFO:sockeye.training:__call__] E=1 B=7000	s/sec=1994.80 tok/sec=110562.08 u/sec=24.21	ppl=13.482153 
[2023-12-17:11:27:08:INFO:sockeye.training:__call__] E=1 B=7050	s/sec=2147.21 tok/sec=106198.05 u/sec=24.71	ppl=13.488930 
[2023-12-17:11:27:10:INFO:sockeye.training:__call__] E=1 B=7100	s/sec=2259.36 tok/sec=105939.81 u/sec=24.82	ppl=13.502704 
[2023-12-17:11:27:12:INFO:sockeye.training:__call__] E=1 B=7150	s/sec=2085.02 tok/sec=108800.35 u/sec=24.68	ppl=13.515030 
[2023-12-17:11:27:14:INFO:sockeye.training:__call__] E=1 B=7200	s/sec=2266.29 tok/sec=106920.82 u/sec=25.07	ppl=13.520870 
[2023-12-17:11:27:16:INFO:sockeye.training:__call__] E=1 B=7250	s/sec=2286.53 tok/sec=106683.70 u/sec=24.72	ppl=13.538700 
[2023-12-17:11:27:18:INFO:sockeye.training:__call__] E=1 B=7300	s/sec=2118.74 tok/sec=107977.32 u/sec=24.52	ppl=13.547332 
[2023-12-17:11:27:20:INFO:sockeye.training:__call__] E=1 B=7350	s/sec=2337.78 tok/sec=106107.89 u/sec=25.11	ppl=13.551338 
[2023-12-17:11:27:22:INFO:sockeye.training:__call__] E=1 B=7400	s/sec=2264.51 tok/sec=105342.72 u/sec=25.69	ppl=13.563710 
[2023-12-17:11:27:24:INFO:sockeye.training:__call__] E=1 B=7450	s/sec=2442.06 tok/sec=103292.74 u/sec=26.09	ppl=13.571934 
[2023-12-17:11:27:26:INFO:sockeye.training:__call__] E=1 B=7500	s/sec=2059.74 tok/sec=108437.43 u/sec=24.76	ppl=13.583101 
[2023-12-17:11:27:28:INFO:sockeye.training:__call__] E=1 B=7550	s/sec=2215.62 tok/sec=106970.18 u/sec=25.50	ppl=13.592012 
[2023-12-17:11:27:30:INFO:sockeye.training:__call__] E=1 B=7600	s/sec=2018.45 tok/sec=111973.25 u/sec=24.35	ppl=13.599093 
[2023-12-17:11:27:32:INFO:sockeye.training:__call__] E=1 B=7650	s/sec=1959.23 tok/sec=110060.23 u/sec=24.39	ppl=13.609347 
[2023-12-17:11:27:34:INFO:sockeye.training:__call__] E=1 B=7700	s/sec=2165.70 tok/sec=108624.43 u/sec=24.66	ppl=13.616538 
[2023-12-17:11:27:36:INFO:sockeye.training:__call__] E=1 B=7750	s/sec=2358.10 tok/sec=103258.20 u/sec=25.11	ppl=13.623370 
[2023-12-17:11:27:38:INFO:sockeye.training:__call__] E=1 B=7800	s/sec=1930.17 tok/sec=113779.90 u/sec=24.03	ppl=13.625021 
[2023-12-17:11:27:40:INFO:sockeye.training:__call__] E=1 B=7850	s/sec=2391.17 tok/sec=105260.55 u/sec=25.63	ppl=13.633094 
[2023-12-17:11:27:42:INFO:sockeye.training:__call__] E=1 B=7900	s/sec=2442.33 tok/sec=103266.13 u/sec=25.92	ppl=13.641049 
[2023-12-17:11:27:44:INFO:sockeye.training:__call__] E=1 B=7950	s/sec=1936.55 tok/sec=111458.24 u/sec=24.26	ppl=13.646912 
[2023-12-17:11:27:46:INFO:sockeye.training:__call__] E=2 B=8000	s/sec=1856.21 tok/sec=103831.14 u/sec=22.75	ppl=13.640558 
[2023-12-17:11:27:46:INFO:sockeye.training:_create_checkpoint] Checkpoint [2]	Updates=8000 Epoch=2 Samples=712944 Time-cost=160.105 Updates/sec=24.984
[2023-12-17:11:27:46:INFO:sockeye.training:_create_checkpoint] Checkpoint [2]	Train-ppl=13.640558
[2023-12-17:11:27:49:INFO:sockeye.training:_evaluate] Checkpoint [2]	Validation-ppl=23.475334	Validation-bleu=0.151654	Validation-chrf=30.600195	Validation-rouge1=0.160526	Validation-rouge2=0.070579	Validation-rougel=0.152247	Validation-length-ratio-mse=1.298197	Validation-ter=96.822121	Validation-avg-sec-per-sent=0.003334	Validation-decode-walltime=1.667208
[2023-12-17:11:27:49:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 30.600195 (delta=1.832007).
[2023-12-17:11:27:49:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [2]	Learning-rate=0.000200
[2023-12-17:11:27:49:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:27:49:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:27:50:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00002"
[2023-12-17:11:27:50:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00002'
[2023-12-17:11:27:50:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:11:27:50:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl'
[2023-12-17:11:27:52:INFO:sockeye.training:__call__] E=2 B=8050	s/sec=731.32 tok/sec=34585.05 u/sec=8.43	ppl=10.624878 
[2023-12-17:11:27:54:INFO:sockeye.training:__call__] E=2 B=8100	s/sec=2057.05 tok/sec=106232.86 u/sec=24.92	ppl=10.683971 
[2023-12-17:11:27:56:INFO:sockeye.training:__call__] E=2 B=8150	s/sec=2110.15 tok/sec=108064.00 u/sec=23.85	ppl=10.698346 
[2023-12-17:11:27:58:INFO:sockeye.training:__call__] E=2 B=8200	s/sec=2116.70 tok/sec=107583.65 u/sec=24.73	ppl=10.732954 
[2023-12-17:11:28:00:INFO:sockeye.training:__call__] E=2 B=8250	s/sec=2063.84 tok/sec=106158.31 u/sec=24.52	ppl=10.668729 
[2023-12-17:11:28:02:INFO:sockeye.training:__call__] E=2 B=8300	s/sec=2365.75 tok/sec=97638.56 u/sec=25.28	ppl=10.723327 
[2023-12-17:11:28:04:INFO:sockeye.training:__call__] E=2 B=8350	s/sec=2321.86 tok/sec=105780.51 u/sec=25.02	ppl=10.802803 
[2023-12-17:11:28:06:INFO:sockeye.training:__call__] E=2 B=8400	s/sec=2380.80 tok/sec=106433.92 u/sec=25.22	ppl=10.840915 
[2023-12-17:11:28:08:INFO:sockeye.training:__call__] E=2 B=8450	s/sec=2452.00 tok/sec=105386.56 u/sec=25.71	ppl=10.882255 
[2023-12-17:11:28:10:INFO:sockeye.training:__call__] E=2 B=8500	s/sec=2098.47 tok/sec=105825.20 u/sec=24.29	ppl=10.880357 
[2023-12-17:11:28:12:INFO:sockeye.training:__call__] E=2 B=8550	s/sec=2111.18 tok/sec=110815.85 u/sec=24.12	ppl=10.937135 
[2023-12-17:11:28:14:INFO:sockeye.training:__call__] E=2 B=8600	s/sec=2064.92 tok/sec=111741.13 u/sec=24.54	ppl=10.945506 
[2023-12-17:11:28:16:INFO:sockeye.training:__call__] E=2 B=8650	s/sec=2151.12 tok/sec=107763.80 u/sec=24.99	ppl=10.957841 
[2023-12-17:11:28:18:INFO:sockeye.training:__call__] E=2 B=8700	s/sec=2116.04 tok/sec=109841.47 u/sec=25.05	ppl=10.995890 
[2023-12-17:11:28:20:INFO:sockeye.training:__call__] E=2 B=8750	s/sec=1988.00 tok/sec=108127.93 u/sec=24.75	ppl=11.008866 
[2023-12-17:11:28:22:INFO:sockeye.training:__call__] E=2 B=8800	s/sec=2227.56 tok/sec=109711.15 u/sec=25.04	ppl=11.021114 
[2023-12-17:11:28:24:INFO:sockeye.training:__call__] E=2 B=8850	s/sec=2085.81 tok/sec=107573.98 u/sec=24.78	ppl=11.017539 
[2023-12-17:11:28:26:INFO:sockeye.training:__call__] E=2 B=8900	s/sec=2276.92 tok/sec=101588.25 u/sec=24.37	ppl=11.061331 
[2023-12-17:11:28:28:INFO:sockeye.training:__call__] E=2 B=8950	s/sec=2016.83 tok/sec=103298.94 u/sec=24.15	ppl=11.068868 
[2023-12-17:11:28:31:INFO:sockeye.training:__call__] E=2 B=9000	s/sec=1810.27 tok/sec=119595.64 u/sec=23.00	ppl=11.079225 
[2023-12-17:11:28:33:INFO:sockeye.training:__call__] E=2 B=9050	s/sec=1890.19 tok/sec=110953.95 u/sec=24.46	ppl=11.095459 
[2023-12-17:11:28:35:INFO:sockeye.training:__call__] E=2 B=9100	s/sec=2358.98 tok/sec=106566.42 u/sec=25.03	ppl=11.130484 
[2023-12-17:11:28:37:INFO:sockeye.training:__call__] E=2 B=9150	s/sec=2443.88 tok/sec=103770.99 u/sec=25.54	ppl=11.152482 
[2023-12-17:11:28:39:INFO:sockeye.training:__call__] E=2 B=9200	s/sec=2239.02 tok/sec=108776.81 u/sec=24.25	ppl=11.179293 
[2023-12-17:11:28:41:INFO:sockeye.training:__call__] E=2 B=9250	s/sec=2020.30 tok/sec=110638.19 u/sec=24.71	ppl=11.187665 
[2023-12-17:11:28:43:INFO:sockeye.training:__call__] E=2 B=9300	s/sec=2504.85 tok/sec=101852.82 u/sec=25.92	ppl=11.230388 
[2023-12-17:11:28:45:INFO:sockeye.training:__call__] E=2 B=9350	s/sec=2532.65 tok/sec=101138.95 u/sec=26.12	ppl=11.269437 
[2023-12-17:11:28:47:INFO:sockeye.training:__call__] E=2 B=9400	s/sec=2109.82 tok/sec=106220.73 u/sec=24.79	ppl=11.285660 
[2023-12-17:11:28:49:INFO:sockeye.training:__call__] E=2 B=9450	s/sec=2352.35 tok/sec=103976.26 u/sec=25.48	ppl=11.304235 
[2023-12-17:11:28:51:INFO:sockeye.training:__call__] E=2 B=9500	s/sec=2252.21 tok/sec=106954.06 u/sec=25.00	ppl=11.329055 
[2023-12-17:11:28:53:INFO:sockeye.training:__call__] E=2 B=9550	s/sec=2314.67 tok/sec=105568.56 u/sec=25.29	ppl=11.350572 
[2023-12-17:11:28:54:INFO:sockeye.training:__call__] E=2 B=9600	s/sec=2559.26 tok/sec=104394.79 u/sec=25.51	ppl=11.372803 
[2023-12-17:11:28:56:INFO:sockeye.training:__call__] E=2 B=9650	s/sec=2106.31 tok/sec=107381.79 u/sec=25.03	ppl=11.374411 
[2023-12-17:11:28:59:INFO:sockeye.training:__call__] E=2 B=9700	s/sec=2178.18 tok/sec=108940.69 u/sec=24.66	ppl=11.400844 
[2023-12-17:11:29:01:INFO:sockeye.training:__call__] E=2 B=9750	s/sec=2112.51 tok/sec=111062.08 u/sec=24.27	ppl=11.405187 
[2023-12-17:11:29:03:INFO:sockeye.training:__call__] E=2 B=9800	s/sec=2368.40 tok/sec=106006.94 u/sec=25.30	ppl=11.414263 
[2023-12-17:11:29:05:INFO:sockeye.training:__call__] E=2 B=9850	s/sec=2060.15 tok/sec=108948.61 u/sec=24.95	ppl=11.420029 
[2023-12-17:11:29:07:INFO:sockeye.training:__call__] E=2 B=9900	s/sec=2293.86 tok/sec=109103.12 u/sec=24.47	ppl=11.433916 
[2023-12-17:11:29:09:INFO:sockeye.training:__call__] E=2 B=9950	s/sec=2181.97 tok/sec=108826.63 u/sec=24.98	ppl=11.449025 
[2023-12-17:11:29:11:INFO:sockeye.training:__call__] E=2 B=10000	s/sec=2090.42 tok/sec=107652.50 u/sec=24.84	ppl=11.464791 
[2023-12-17:11:29:13:INFO:sockeye.training:__call__] E=2 B=10050	s/sec=2503.08 tok/sec=102559.52 u/sec=26.07	ppl=11.476800 
[2023-12-17:11:29:15:INFO:sockeye.training:__call__] E=2 B=10100	s/sec=2110.38 tok/sec=110670.72 u/sec=24.65	ppl=11.488621 
[2023-12-17:11:29:17:INFO:sockeye.training:__call__] E=2 B=10150	s/sec=2312.92 tok/sec=101366.33 u/sec=24.17	ppl=11.509876 
[2023-12-17:11:29:19:INFO:sockeye.training:__call__] E=2 B=10200	s/sec=2186.53 tok/sec=109906.97 u/sec=24.19	ppl=11.525590 
[2023-12-17:11:29:21:INFO:sockeye.training:__call__] E=2 B=10250	s/sec=2577.72 tok/sec=103599.81 u/sec=25.57	ppl=11.547100 
[2023-12-17:11:29:23:INFO:sockeye.training:__call__] E=2 B=10300	s/sec=2261.55 tok/sec=107230.35 u/sec=24.62	ppl=11.566534 
[2023-12-17:11:29:25:INFO:sockeye.training:__call__] E=2 B=10350	s/sec=2280.10 tok/sec=106447.49 u/sec=25.18	ppl=11.578456 
[2023-12-17:11:29:27:INFO:sockeye.training:__call__] E=2 B=10400	s/sec=2435.08 tok/sec=105651.38 u/sec=25.07	ppl=11.598118 
[2023-12-17:11:29:29:INFO:sockeye.training:__call__] E=2 B=10450	s/sec=2315.60 tok/sec=105251.41 u/sec=25.04	ppl=11.613501 
[2023-12-17:11:29:31:INFO:sockeye.training:__call__] E=2 B=10500	s/sec=2320.71 tok/sec=103493.92 u/sec=25.72	ppl=11.623477 
[2023-12-17:11:29:33:INFO:sockeye.training:__call__] E=2 B=10550	s/sec=2258.98 tok/sec=107075.48 u/sec=25.21	ppl=11.635832 
[2023-12-17:11:29:35:INFO:sockeye.training:__call__] E=2 B=10600	s/sec=2341.43 tok/sec=107777.77 u/sec=25.02	ppl=11.655678 
[2023-12-17:11:29:37:INFO:sockeye.training:__call__] E=2 B=10650	s/sec=2082.71 tok/sec=106569.52 u/sec=25.52	ppl=11.671093 
[2023-12-17:11:29:39:INFO:sockeye.training:__call__] E=2 B=10700	s/sec=2395.14 tok/sec=105516.56 u/sec=25.46	ppl=11.686696 
[2023-12-17:11:29:41:INFO:sockeye.training:__call__] E=2 B=10750	s/sec=2087.06 tok/sec=111828.91 u/sec=24.34	ppl=11.701229 
[2023-12-17:11:29:43:INFO:sockeye.training:__call__] E=2 B=10800	s/sec=2522.72 tok/sec=104119.53 u/sec=25.60	ppl=11.723277 
[2023-12-17:11:29:45:INFO:sockeye.training:__call__] E=2 B=10850	s/sec=2172.57 tok/sec=105022.09 u/sec=25.10	ppl=11.731202 
[2023-12-17:11:29:46:INFO:sockeye.training:__call__] E=2 B=10900	s/sec=2158.86 tok/sec=107513.44 u/sec=25.31	ppl=11.742483 
[2023-12-17:11:29:48:INFO:sockeye.training:__call__] E=2 B=10950	s/sec=2597.62 tok/sec=104199.48 u/sec=25.57	ppl=11.755462 
[2023-12-17:11:29:50:INFO:sockeye.training:__call__] E=2 B=11000	s/sec=2187.46 tok/sec=107104.31 u/sec=25.32	ppl=11.761981 
[2023-12-17:11:29:52:INFO:sockeye.training:__call__] E=2 B=11050	s/sec=2229.42 tok/sec=108401.51 u/sec=25.24	ppl=11.772326 
[2023-12-17:11:29:54:INFO:sockeye.training:__call__] E=2 B=11100	s/sec=2171.52 tok/sec=102292.43 u/sec=24.06	ppl=11.785421 
[2023-12-17:11:29:56:INFO:sockeye.training:__call__] E=2 B=11150	s/sec=2182.69 tok/sec=102864.25 u/sec=24.80	ppl=11.799474 
[2023-12-17:11:29:58:INFO:sockeye.training:__call__] E=2 B=11200	s/sec=2542.34 tok/sec=103459.52 u/sec=25.55	ppl=11.811902 
[2023-12-17:11:30:00:INFO:sockeye.training:__call__] E=2 B=11250	s/sec=2209.17 tok/sec=106972.04 u/sec=25.10	ppl=11.826524 
[2023-12-17:11:30:02:INFO:sockeye.training:__call__] E=2 B=11300	s/sec=2030.25 tok/sec=111976.12 u/sec=24.40	ppl=11.841049 
[2023-12-17:11:30:05:INFO:sockeye.training:__call__] E=2 B=11350	s/sec=2227.00 tok/sec=106423.03 u/sec=24.63	ppl=11.850221 
[2023-12-17:11:30:07:INFO:sockeye.training:__call__] E=2 B=11400	s/sec=2119.23 tok/sec=105734.43 u/sec=24.48	ppl=11.856158 
[2023-12-17:11:30:09:INFO:sockeye.training:__call__] E=2 B=11450	s/sec=2010.59 tok/sec=109376.23 u/sec=24.64	ppl=11.864036 
[2023-12-17:11:30:11:INFO:sockeye.training:__call__] E=2 B=11500	s/sec=2191.65 tok/sec=108523.97 u/sec=24.50	ppl=11.876063 
[2023-12-17:11:30:13:INFO:sockeye.training:__call__] E=2 B=11550	s/sec=2237.70 tok/sec=106345.41 u/sec=25.20	ppl=11.885707 
[2023-12-17:11:30:15:INFO:sockeye.training:__call__] E=2 B=11600	s/sec=2499.67 tok/sec=100906.34 u/sec=25.65	ppl=11.895304 
[2023-12-17:11:30:17:INFO:sockeye.training:__call__] E=2 B=11650	s/sec=2032.51 tok/sec=109388.73 u/sec=24.91	ppl=11.901931 
[2023-12-17:11:30:19:INFO:sockeye.training:__call__] E=2 B=11700	s/sec=2222.58 tok/sec=108381.97 u/sec=25.03	ppl=11.912242 
[2023-12-17:11:30:21:INFO:sockeye.training:__call__] E=2 B=11750	s/sec=2094.99 tok/sec=105985.54 u/sec=25.08	ppl=11.918556 
[2023-12-17:11:30:23:INFO:sockeye.training:__call__] E=2 B=11800	s/sec=2104.67 tok/sec=110033.94 u/sec=24.96	ppl=11.929576 
[2023-12-17:11:30:25:INFO:sockeye.training:__call__] E=2 B=11850	s/sec=2220.80 tok/sec=106630.60 u/sec=25.19	ppl=11.936036 
[2023-12-17:11:30:27:INFO:sockeye.training:__call__] E=2 B=11900	s/sec=2192.94 tok/sec=106281.27 u/sec=25.71	ppl=11.943710 
[2023-12-17:11:30:28:INFO:sockeye.training:__call__] E=2 B=11950	s/sec=2365.47 tok/sec=103998.32 u/sec=25.80	ppl=11.952154 
[2023-12-17:11:30:31:INFO:sockeye.training:__call__] E=3 B=12000	s/sec=2043.45 tok/sec=101610.43 u/sec=23.74	ppl=11.946895 
[2023-12-17:11:30:31:INFO:sockeye.training:_create_checkpoint] Checkpoint [3]	Updates=12000 Epoch=3 Samples=1069512 Time-cost=160.446 Updates/sec=24.931
[2023-12-17:11:30:31:INFO:sockeye.training:_create_checkpoint] Checkpoint [3]	Train-ppl=11.946895
[2023-12-17:11:30:35:INFO:sockeye.training:_evaluate] Checkpoint [3]	Validation-ppl=24.198217	Validation-bleu=0.146511	Validation-chrf=30.119190	Validation-rouge1=0.160281	Validation-rouge2=0.078863	Validation-rougel=0.152774	Validation-length-ratio-mse=1.283743	Validation-ter=95.914156	Validation-avg-sec-per-sent=0.006637	Validation-decode-walltime=3.318676
[2023-12-17:11:30:35:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 1 checkpoints, best so far: 30.600195
[2023-12-17:11:30:35:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [3]	Learning-rate=0.000200
[2023-12-17:11:30:35:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:30:35:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:30:36:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00003"
[2023-12-17:11:30:38:INFO:sockeye.training:__call__] E=3 B=12050	s/sec=671.04 tok/sec=28916.04 u/sec=7.11	ppl=10.016674 
[2023-12-17:11:30:40:INFO:sockeye.training:__call__] E=3 B=12100	s/sec=2178.73 tok/sec=106676.94 u/sec=25.22	ppl=9.851505 
[2023-12-17:11:30:42:INFO:sockeye.training:__call__] E=3 B=12150	s/sec=2546.20 tok/sec=102921.63 u/sec=25.42	ppl=9.905411 
[2023-12-17:11:30:44:INFO:sockeye.training:__call__] E=3 B=12200	s/sec=1908.69 tok/sec=112826.28 u/sec=24.30	ppl=9.866644 
[2023-12-17:11:30:46:INFO:sockeye.training:__call__] E=3 B=12250	s/sec=2087.81 tok/sec=108089.58 u/sec=24.81	ppl=9.814334 
[2023-12-17:11:30:48:INFO:sockeye.training:__call__] E=3 B=12300	s/sec=2244.14 tok/sec=107525.81 u/sec=25.09	ppl=9.826744 
[2023-12-17:11:30:50:INFO:sockeye.training:__call__] E=3 B=12350	s/sec=2224.54 tok/sec=107265.91 u/sec=25.42	ppl=9.843311 
[2023-12-17:11:30:52:INFO:sockeye.training:__call__] E=3 B=12400	s/sec=2307.28 tok/sec=105172.61 u/sec=25.48	ppl=9.872836 
[2023-12-17:11:30:54:INFO:sockeye.training:__call__] E=3 B=12450	s/sec=2060.99 tok/sec=108959.27 u/sec=24.72	ppl=9.870092 
[2023-12-17:11:30:56:INFO:sockeye.training:__call__] E=3 B=12500	s/sec=2393.78 tok/sec=102199.22 u/sec=25.31	ppl=9.890979 
[2023-12-17:11:30:58:INFO:sockeye.training:__call__] E=3 B=12550	s/sec=2133.91 tok/sec=110667.71 u/sec=24.29	ppl=9.915985 
[2023-12-17:11:31:00:INFO:sockeye.training:__call__] E=3 B=12600	s/sec=2212.62 tok/sec=104641.94 u/sec=25.99	ppl=9.911805 
[2023-12-17:11:31:02:INFO:sockeye.training:__call__] E=3 B=12650	s/sec=2110.78 tok/sec=109792.25 u/sec=24.89	ppl=9.940040 
[2023-12-17:11:31:04:INFO:sockeye.training:__call__] E=3 B=12700	s/sec=2235.94 tok/sec=107325.16 u/sec=25.22	ppl=9.959882 
[2023-12-17:11:31:06:INFO:sockeye.training:__call__] E=3 B=12750	s/sec=2074.89 tok/sec=108352.51 u/sec=24.70	ppl=9.970109 
[2023-12-17:11:31:08:INFO:sockeye.training:__call__] E=3 B=12800	s/sec=2250.92 tok/sec=107172.94 u/sec=25.21	ppl=9.985772 
[2023-12-17:11:31:09:INFO:sockeye.training:__call__] E=3 B=12850	s/sec=2236.71 tok/sec=105536.80 u/sec=25.46	ppl=9.982103 
[2023-12-17:11:31:12:INFO:sockeye.training:__call__] E=3 B=12900	s/sec=2181.94 tok/sec=109938.80 u/sec=24.35	ppl=10.005415 
[2023-12-17:11:31:14:INFO:sockeye.training:__call__] E=3 B=12950	s/sec=2174.26 tok/sec=109760.33 u/sec=24.80	ppl=10.029045 
[2023-12-17:11:31:16:INFO:sockeye.training:__call__] E=3 B=13000	s/sec=2437.45 tok/sec=104041.20 u/sec=25.31	ppl=10.071231 
[2023-12-17:11:31:18:INFO:sockeye.training:__call__] E=3 B=13050	s/sec=2356.85 tok/sec=105280.59 u/sec=25.44	ppl=10.100793 
[2023-12-17:11:31:19:INFO:sockeye.training:__call__] E=3 B=13100	s/sec=2450.26 tok/sec=105785.71 u/sec=25.52	ppl=10.140637 
[2023-12-17:11:31:21:INFO:sockeye.training:__call__] E=3 B=13150	s/sec=2143.04 tok/sec=109921.66 u/sec=24.94	ppl=10.164660 
[2023-12-17:11:31:24:INFO:sockeye.training:__call__] E=3 B=13200	s/sec=2218.29 tok/sec=113552.07 u/sec=23.82	ppl=10.170193 
[2023-12-17:11:31:26:INFO:sockeye.training:__call__] E=3 B=13250	s/sec=2264.51 tok/sec=109640.12 u/sec=24.57	ppl=10.190507 
[2023-12-17:11:31:28:INFO:sockeye.training:__call__] E=3 B=13300	s/sec=2358.67 tok/sec=106095.56 u/sec=25.29	ppl=10.215236 
[2023-12-17:11:31:29:INFO:sockeye.training:__call__] E=3 B=13350	s/sec=2327.31 tok/sec=100931.62 u/sec=26.40	ppl=10.235486 
[2023-12-17:11:31:31:INFO:sockeye.training:__call__] E=3 B=13400	s/sec=2276.14 tok/sec=108174.16 u/sec=24.83	ppl=10.247119 
[2023-12-17:11:31:34:INFO:sockeye.training:__call__] E=3 B=13450	s/sec=2003.16 tok/sec=106876.20 u/sec=23.94	ppl=10.256496 
[2023-12-17:11:31:36:INFO:sockeye.training:__call__] E=3 B=13500	s/sec=2200.78 tok/sec=108314.83 u/sec=24.61	ppl=10.277170 
[2023-12-17:11:31:38:INFO:sockeye.training:__call__] E=3 B=13550	s/sec=2439.04 tok/sec=105450.13 u/sec=25.16	ppl=10.296552 
[2023-12-17:11:31:40:INFO:sockeye.training:__call__] E=3 B=13600	s/sec=2250.96 tok/sec=107448.05 u/sec=24.60	ppl=10.318140 
[2023-12-17:11:31:42:INFO:sockeye.training:__call__] E=3 B=13650	s/sec=2272.71 tok/sec=107506.46 u/sec=24.75	ppl=10.335599 
[2023-12-17:11:31:44:INFO:sockeye.training:__call__] E=3 B=13700	s/sec=2138.68 tok/sec=108317.60 u/sec=24.57	ppl=10.347753 
[2023-12-17:11:31:46:INFO:sockeye.training:__call__] E=3 B=13750	s/sec=1978.96 tok/sec=109222.73 u/sec=24.49	ppl=10.354556 
[2023-12-17:11:31:48:INFO:sockeye.training:__call__] E=3 B=13800	s/sec=2180.03 tok/sec=107312.39 u/sec=25.14	ppl=10.366620 
[2023-12-17:11:31:50:INFO:sockeye.training:__call__] E=3 B=13850	s/sec=2556.85 tok/sec=101737.39 u/sec=26.20	ppl=10.385139 
[2023-12-17:11:31:52:INFO:sockeye.training:__call__] E=3 B=13900	s/sec=2069.07 tok/sec=109037.31 u/sec=24.82	ppl=10.390931 
[2023-12-17:11:31:54:INFO:sockeye.training:__call__] E=3 B=13950	s/sec=2329.46 tok/sec=105439.35 u/sec=25.41	ppl=10.411995 
[2023-12-17:11:31:56:INFO:sockeye.training:__call__] E=3 B=14000	s/sec=2081.09 tok/sec=109552.37 u/sec=24.27	ppl=10.424477 
[2023-12-17:11:31:58:INFO:sockeye.training:__call__] E=3 B=14050	s/sec=2264.76 tok/sec=109382.01 u/sec=25.05	ppl=10.435299 
[2023-12-17:11:32:00:INFO:sockeye.training:__call__] E=3 B=14100	s/sec=2431.80 tok/sec=104749.55 u/sec=25.29	ppl=10.444945 
[2023-12-17:11:32:02:INFO:sockeye.training:__call__] E=3 B=14150	s/sec=2283.66 tok/sec=104959.59 u/sec=25.26	ppl=10.465674 
[2023-12-17:11:32:04:INFO:sockeye.training:__call__] E=3 B=14200	s/sec=2393.00 tok/sec=105421.70 u/sec=25.35	ppl=10.477284 
[2023-12-17:11:32:06:INFO:sockeye.training:__call__] E=3 B=14250	s/sec=2561.12 tok/sec=101845.96 u/sec=25.99	ppl=10.493326 
[2023-12-17:11:32:08:INFO:sockeye.training:__call__] E=3 B=14300	s/sec=2313.67 tok/sec=103807.75 u/sec=25.28	ppl=10.502439 
[2023-12-17:11:32:10:INFO:sockeye.training:__call__] E=3 B=14350	s/sec=2222.34 tok/sec=105872.80 u/sec=24.98	ppl=10.511258 
[2023-12-17:11:32:12:INFO:sockeye.training:__call__] E=3 B=14400	s/sec=2321.13 tok/sec=106058.53 u/sec=25.06	ppl=10.531826 
[2023-12-17:11:32:14:INFO:sockeye.training:__call__] E=3 B=14450	s/sec=2309.55 tok/sec=106851.30 u/sec=24.84	ppl=10.542439 
[2023-12-17:11:32:16:INFO:sockeye.training:__call__] E=3 B=14500	s/sec=2196.21 tok/sec=108075.51 u/sec=24.42	ppl=10.554888 
[2023-12-17:11:32:18:INFO:sockeye.training:__call__] E=3 B=14550	s/sec=2279.63 tok/sec=106484.72 u/sec=25.22	ppl=10.566321 
[2023-12-17:11:32:20:INFO:sockeye.training:__call__] E=3 B=14600	s/sec=2005.44 tok/sec=110112.79 u/sec=24.82	ppl=10.576395 
[2023-12-17:11:32:22:INFO:sockeye.training:__call__] E=3 B=14650	s/sec=2224.40 tok/sec=105067.29 u/sec=25.60	ppl=10.584380 
[2023-12-17:11:32:23:INFO:sockeye.training:__call__] E=3 B=14700	s/sec=2426.77 tok/sec=103974.20 u/sec=25.32	ppl=10.597297 
[2023-12-17:11:32:25:INFO:sockeye.training:__call__] E=3 B=14750	s/sec=2254.17 tok/sec=107516.77 u/sec=25.43	ppl=10.606294 
[2023-12-17:11:32:27:INFO:sockeye.training:__call__] E=3 B=14800	s/sec=2248.81 tok/sec=107012.97 u/sec=25.05	ppl=10.617163 
[2023-12-17:11:32:30:INFO:sockeye.training:__call__] E=3 B=14850	s/sec=1840.48 tok/sec=112800.46 u/sec=24.06	ppl=10.619899 
[2023-12-17:11:32:32:INFO:sockeye.training:__call__] E=3 B=14900	s/sec=2257.81 tok/sec=104153.04 u/sec=24.80	ppl=10.630864 
[2023-12-17:11:32:34:INFO:sockeye.training:__call__] E=3 B=14950	s/sec=2068.74 tok/sec=109071.79 u/sec=24.63	ppl=10.642694 
[2023-12-17:11:32:36:INFO:sockeye.training:__call__] E=3 B=15000	s/sec=2055.46 tok/sec=108636.51 u/sec=24.90	ppl=10.653150 
[2023-12-17:11:32:38:INFO:sockeye.training:__call__] E=3 B=15050	s/sec=2390.22 tok/sec=106466.68 u/sec=25.02	ppl=10.665041 
[2023-12-17:11:32:40:INFO:sockeye.training:__call__] E=3 B=15100	s/sec=2171.52 tok/sec=105069.22 u/sec=25.13	ppl=10.678170 
[2023-12-17:11:32:42:INFO:sockeye.training:__call__] E=3 B=15150	s/sec=2199.52 tok/sec=107392.91 u/sec=24.46	ppl=10.690890 
[2023-12-17:11:32:44:INFO:sockeye.training:__call__] E=3 B=15200	s/sec=2193.95 tok/sec=99490.46 u/sec=24.44	ppl=10.701946 
[2023-12-17:11:32:46:INFO:sockeye.training:__call__] E=3 B=15250	s/sec=2205.93 tok/sec=107326.08 u/sec=25.02	ppl=10.705835 
[2023-12-17:11:32:48:INFO:sockeye.training:__call__] E=3 B=15300	s/sec=2051.47 tok/sec=107509.66 u/sec=25.04	ppl=10.709276 
[2023-12-17:11:32:50:INFO:sockeye.training:__call__] E=3 B=15350	s/sec=2472.70 tok/sec=105044.72 u/sec=25.50	ppl=10.721957 
[2023-12-17:11:32:52:INFO:sockeye.training:__call__] E=3 B=15400	s/sec=2237.05 tok/sec=109196.05 u/sec=24.49	ppl=10.736963 
[2023-12-17:11:32:54:INFO:sockeye.training:__call__] E=3 B=15450	s/sec=2032.86 tok/sec=109573.99 u/sec=25.06	ppl=10.747530 
[2023-12-17:11:32:56:INFO:sockeye.training:__call__] E=3 B=15500	s/sec=2328.54 tok/sec=104286.47 u/sec=25.09	ppl=10.759964 
[2023-12-17:11:32:58:INFO:sockeye.training:__call__] E=3 B=15550	s/sec=2141.15 tok/sec=109095.80 u/sec=24.69	ppl=10.768632 
[2023-12-17:11:33:00:INFO:sockeye.training:__call__] E=3 B=15600	s/sec=2021.96 tok/sec=108612.50 u/sec=24.88	ppl=10.776463 
[2023-12-17:11:33:02:INFO:sockeye.training:__call__] E=3 B=15650	s/sec=2178.40 tok/sec=106566.67 u/sec=24.84	ppl=10.785695 
[2023-12-17:11:33:04:INFO:sockeye.training:__call__] E=3 B=15700	s/sec=2110.38 tok/sec=108138.44 u/sec=24.29	ppl=10.795495 
[2023-12-17:11:33:06:INFO:sockeye.training:__call__] E=3 B=15750	s/sec=2121.62 tok/sec=102053.13 u/sec=24.02	ppl=10.803107 
[2023-12-17:11:33:08:INFO:sockeye.training:__call__] E=3 B=15800	s/sec=2332.58 tok/sec=102808.23 u/sec=24.84	ppl=10.812838 
[2023-12-17:11:33:10:INFO:sockeye.training:__call__] E=3 B=15850	s/sec=2311.72 tok/sec=105898.54 u/sec=25.04	ppl=10.821610 
[2023-12-17:11:33:12:INFO:sockeye.training:__call__] E=3 B=15900	s/sec=2123.30 tok/sec=107386.35 u/sec=25.28	ppl=10.827376 
[2023-12-17:11:33:14:INFO:sockeye.training:__call__] E=3 B=15950	s/sec=2114.06 tok/sec=108718.46 u/sec=24.93	ppl=10.832222 
[2023-12-17:11:33:16:INFO:sockeye.training:__call__] E=4 B=16000	s/sec=1899.84 tok/sec=102458.06 u/sec=23.10	ppl=10.824535 
[2023-12-17:11:33:16:INFO:sockeye.training:_create_checkpoint] Checkpoint [4]	Updates=16000 Epoch=4 Samples=1425712 Time-cost=160.375 Updates/sec=24.942
[2023-12-17:11:33:16:INFO:sockeye.training:_create_checkpoint] Checkpoint [4]	Train-ppl=10.824535
[2023-12-17:11:33:19:INFO:sockeye.training:_evaluate] Checkpoint [4]	Validation-ppl=25.573629	Validation-bleu=0.157214	Validation-chrf=30.918991	Validation-rouge1=0.172288	Validation-rouge2=0.084025	Validation-rougel=0.165478	Validation-length-ratio-mse=1.274053	Validation-ter=96.904664	Validation-avg-sec-per-sent=0.003402	Validation-decode-walltime=1.700753
[2023-12-17:11:33:19:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 30.918991 (delta=0.318796).
[2023-12-17:11:33:19:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [4]	Learning-rate=0.000200
[2023-12-17:11:33:19:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:33:19:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:33:19:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00004"
[2023-12-17:11:33:19:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00004'
[2023-12-17:11:33:20:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:11:33:20:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl'
[2023-12-17:11:33:22:INFO:sockeye.training:__call__] E=4 B=16050	s/sec=771.14 tok/sec=35202.42 u/sec=8.53	ppl=8.917675 
[2023-12-17:11:33:24:INFO:sockeye.training:__call__] E=4 B=16100	s/sec=2252.91 tok/sec=106902.21 u/sec=24.79	ppl=8.946637 
[2023-12-17:11:33:26:INFO:sockeye.training:__call__] E=4 B=16150	s/sec=2149.31 tok/sec=106239.74 u/sec=25.54	ppl=8.860597 
[2023-12-17:11:33:28:INFO:sockeye.training:__call__] E=4 B=16200	s/sec=2430.83 tok/sec=99511.69 u/sec=25.07	ppl=8.916239 
[2023-12-17:11:33:30:INFO:sockeye.training:__call__] E=4 B=16250	s/sec=2223.41 tok/sec=106818.75 u/sec=24.81	ppl=8.925710 
[2023-12-17:11:33:32:INFO:sockeye.training:__call__] E=4 B=16300	s/sec=2122.38 tok/sec=110106.69 u/sec=25.12	ppl=8.944700 
[2023-12-17:11:33:34:INFO:sockeye.training:__call__] E=4 B=16350	s/sec=2452.15 tok/sec=103974.31 u/sec=25.84	ppl=8.987162 
[2023-12-17:11:33:36:INFO:sockeye.training:__call__] E=4 B=16400	s/sec=1736.69 tok/sec=112767.95 u/sec=24.34	ppl=8.970574 
[2023-12-17:11:33:38:INFO:sockeye.training:__call__] E=4 B=16450	s/sec=1954.12 tok/sec=110449.37 u/sec=24.87	ppl=9.003515 
[2023-12-17:11:33:40:INFO:sockeye.training:__call__] E=4 B=16500	s/sec=2162.46 tok/sec=108342.25 u/sec=24.48	ppl=9.039726 
[2023-12-17:11:33:42:INFO:sockeye.training:__call__] E=4 B=16550	s/sec=2173.87 tok/sec=108140.39 u/sec=24.70	ppl=9.041873 
[2023-12-17:11:33:44:INFO:sockeye.training:__call__] E=4 B=16600	s/sec=2093.61 tok/sec=109432.21 u/sec=24.50	ppl=9.060630 
[2023-12-17:11:33:46:INFO:sockeye.training:__call__] E=4 B=16650	s/sec=2324.73 tok/sec=105921.04 u/sec=25.01	ppl=9.081475 
[2023-12-17:11:33:48:INFO:sockeye.training:__call__] E=4 B=16700	s/sec=2272.62 tok/sec=103570.36 u/sec=25.05	ppl=9.097918 
[2023-12-17:11:33:50:INFO:sockeye.training:__call__] E=4 B=16750	s/sec=2388.46 tok/sec=103094.83 u/sec=26.01	ppl=9.112441 
[2023-12-17:11:33:52:INFO:sockeye.training:__call__] E=4 B=16800	s/sec=2313.26 tok/sec=106054.21 u/sec=25.28	ppl=9.150787 
[2023-12-17:11:33:54:INFO:sockeye.training:__call__] E=4 B=16850	s/sec=2462.46 tok/sec=104587.20 u/sec=25.44	ppl=9.186098 
[2023-12-17:11:33:56:INFO:sockeye.training:__call__] E=4 B=16900	s/sec=2522.52 tok/sec=99156.46 u/sec=26.72	ppl=9.227606 
[2023-12-17:11:33:58:INFO:sockeye.training:__call__] E=4 B=16950	s/sec=2129.24 tok/sec=108937.35 u/sec=24.02	ppl=9.252101 
[2023-12-17:11:34:00:INFO:sockeye.training:__call__] E=4 B=17000	s/sec=2309.28 tok/sec=105892.82 u/sec=25.46	ppl=9.271247 
[2023-12-17:11:34:02:INFO:sockeye.training:__call__] E=4 B=17050	s/sec=2225.81 tok/sec=107400.34 u/sec=24.36	ppl=9.290739 
[2023-12-17:11:34:04:INFO:sockeye.training:__call__] E=4 B=17100	s/sec=1813.00 tok/sec=118547.46 u/sec=21.83	ppl=9.300319 
[2023-12-17:11:34:06:INFO:sockeye.training:__call__] E=4 B=17150	s/sec=2293.33 tok/sec=106254.97 u/sec=25.32	ppl=9.318648 
[2023-12-17:11:34:08:INFO:sockeye.training:__call__] E=4 B=17200	s/sec=2198.38 tok/sec=107860.72 u/sec=25.03	ppl=9.341177 
[2023-12-17:11:34:10:INFO:sockeye.training:__call__] E=4 B=17250	s/sec=2093.15 tok/sec=109130.41 u/sec=24.87	ppl=9.348260 
[2023-12-17:11:34:12:INFO:sockeye.training:__call__] E=4 B=17300	s/sec=2104.07 tok/sec=107488.82 u/sec=24.63	ppl=9.352187 
[2023-12-17:11:34:14:INFO:sockeye.training:__call__] E=4 B=17350	s/sec=2103.37 tok/sec=108502.74 u/sec=24.34	ppl=9.371058 
[2023-12-17:11:34:16:INFO:sockeye.training:__call__] E=4 B=17400	s/sec=2152.14 tok/sec=107599.28 u/sec=24.86	ppl=9.381219 
[2023-12-17:11:34:18:INFO:sockeye.training:__call__] E=4 B=17450	s/sec=2277.84 tok/sec=106942.75 u/sec=24.93	ppl=9.395338 
[2023-12-17:11:34:20:INFO:sockeye.training:__call__] E=4 B=17500	s/sec=2250.52 tok/sec=107264.32 u/sec=24.76	ppl=9.411775 
[2023-12-17:11:34:22:INFO:sockeye.training:__call__] E=4 B=17550	s/sec=2683.17 tok/sec=100160.30 u/sec=26.28	ppl=9.434743 
[2023-12-17:11:34:24:INFO:sockeye.training:__call__] E=4 B=17600	s/sec=2273.10 tok/sec=107523.16 u/sec=25.28	ppl=9.448146 
[2023-12-17:11:34:26:INFO:sockeye.training:__call__] E=4 B=17650	s/sec=2487.34 tok/sec=100226.70 u/sec=26.04	ppl=9.465117 
[2023-12-17:11:34:28:INFO:sockeye.training:__call__] E=4 B=17700	s/sec=2091.09 tok/sec=109778.24 u/sec=24.66	ppl=9.473233 
[2023-12-17:11:34:30:INFO:sockeye.training:__call__] E=4 B=17750	s/sec=2241.21 tok/sec=105772.06 u/sec=25.19	ppl=9.489519 
[2023-12-17:11:34:32:INFO:sockeye.training:__call__] E=4 B=17800	s/sec=2160.87 tok/sec=106879.25 u/sec=24.92	ppl=9.503529 
[2023-12-17:11:34:34:INFO:sockeye.training:__call__] E=4 B=17850	s/sec=2279.01 tok/sec=107438.06 u/sec=25.03	ppl=9.521187 
[2023-12-17:11:34:36:INFO:sockeye.training:__call__] E=4 B=17900	s/sec=2720.37 tok/sec=99492.83 u/sec=26.48	ppl=9.539657 
[2023-12-17:11:34:38:INFO:sockeye.training:__call__] E=4 B=17950	s/sec=2338.47 tok/sec=106554.29 u/sec=24.98	ppl=9.554387 
[2023-12-17:11:34:40:INFO:sockeye.training:__call__] E=4 B=18000	s/sec=2347.85 tok/sec=105437.40 u/sec=25.43	ppl=9.572145 
[2023-12-17:11:34:42:INFO:sockeye.training:__call__] E=4 B=18050	s/sec=2391.69 tok/sec=103701.72 u/sec=25.21	ppl=9.590203 
[2023-12-17:11:34:44:INFO:sockeye.training:__call__] E=4 B=18100	s/sec=2168.21 tok/sec=107440.73 u/sec=25.05	ppl=9.610583 
[2023-12-17:11:34:46:INFO:sockeye.training:__call__] E=4 B=18150	s/sec=2042.25 tok/sec=109684.38 u/sec=24.88	ppl=9.623043 
[2023-12-17:11:34:48:INFO:sockeye.training:__call__] E=4 B=18200	s/sec=2175.97 tok/sec=107826.16 u/sec=24.91	ppl=9.635046 
[2023-12-17:11:34:50:INFO:sockeye.training:__call__] E=4 B=18250	s/sec=2500.31 tok/sec=104025.87 u/sec=25.49	ppl=9.652817 
[2023-12-17:11:34:52:INFO:sockeye.training:__call__] E=4 B=18300	s/sec=2249.46 tok/sec=103443.29 u/sec=24.75	ppl=9.663840 
[2023-12-17:11:34:54:INFO:sockeye.training:__call__] E=4 B=18350	s/sec=2199.50 tok/sec=107928.38 u/sec=25.18	ppl=9.674361 
[2023-12-17:11:34:56:INFO:sockeye.training:__call__] E=4 B=18400	s/sec=2177.94 tok/sec=106894.04 u/sec=24.84	ppl=9.687692 
[2023-12-17:11:34:58:INFO:sockeye.training:__call__] E=4 B=18450	s/sec=2563.23 tok/sec=101344.24 u/sec=25.71	ppl=9.701103 
[2023-12-17:11:35:00:INFO:sockeye.training:__call__] E=4 B=18500	s/sec=2248.70 tok/sec=103510.77 u/sec=24.53	ppl=9.712599 
[2023-12-17:11:35:02:INFO:sockeye.training:__call__] E=4 B=18550	s/sec=2169.25 tok/sec=106503.37 u/sec=24.79	ppl=9.722417 
[2023-12-17:11:35:04:INFO:sockeye.training:__call__] E=4 B=18600	s/sec=1880.07 tok/sec=112460.90 u/sec=24.38	ppl=9.729283 
[2023-12-17:11:35:06:INFO:sockeye.training:__call__] E=4 B=18650	s/sec=2252.22 tok/sec=108106.46 u/sec=25.00	ppl=9.738542 
[2023-12-17:11:35:08:INFO:sockeye.training:__call__] E=4 B=18700	s/sec=2211.49 tok/sec=106601.77 u/sec=25.13	ppl=9.749000 
[2023-12-17:11:35:10:INFO:sockeye.training:__call__] E=4 B=18750	s/sec=2462.13 tok/sec=105192.81 u/sec=25.56	ppl=9.762095 
[2023-12-17:11:35:12:INFO:sockeye.training:__call__] E=4 B=18800	s/sec=2268.63 tok/sec=108607.58 u/sec=24.88	ppl=9.774668 
[2023-12-17:11:35:14:INFO:sockeye.training:__call__] E=4 B=18850	s/sec=2135.21 tok/sec=108608.82 u/sec=24.90	ppl=9.781567 
[2023-12-17:11:35:16:INFO:sockeye.training:__call__] E=4 B=18900	s/sec=2491.60 tok/sec=102525.83 u/sec=25.70	ppl=9.797903 
[2023-12-17:11:35:18:INFO:sockeye.training:__call__] E=4 B=18950	s/sec=1971.41 tok/sec=108458.53 u/sec=24.07	ppl=9.807500 
[2023-12-17:11:35:20:INFO:sockeye.training:__call__] E=4 B=19000	s/sec=2266.18 tok/sec=107093.14 u/sec=25.29	ppl=9.816487 
[2023-12-17:11:35:22:INFO:sockeye.training:__call__] E=4 B=19050	s/sec=2052.79 tok/sec=110207.43 u/sec=23.94	ppl=9.828563 
[2023-12-17:11:35:24:INFO:sockeye.training:__call__] E=4 B=19100	s/sec=2240.77 tok/sec=108261.13 u/sec=25.01	ppl=9.839710 
[2023-12-17:11:35:26:INFO:sockeye.training:__call__] E=4 B=19150	s/sec=2068.93 tok/sec=106406.48 u/sec=25.21	ppl=9.843456 
[2023-12-17:11:35:28:INFO:sockeye.training:__call__] E=4 B=19200	s/sec=2198.32 tok/sec=107124.06 u/sec=25.07	ppl=9.852151 
[2023-12-17:11:35:30:INFO:sockeye.training:__call__] E=4 B=19250	s/sec=2096.44 tok/sec=112203.24 u/sec=23.61	ppl=9.860877 
[2023-12-17:11:35:32:INFO:sockeye.training:__call__] E=4 B=19300	s/sec=1930.75 tok/sec=108761.33 u/sec=24.98	ppl=9.868093 
[2023-12-17:11:35:34:INFO:sockeye.training:__call__] E=4 B=19350	s/sec=1906.01 tok/sec=111913.08 u/sec=24.36	ppl=9.874656 
[2023-12-17:11:35:36:INFO:sockeye.training:__call__] E=4 B=19400	s/sec=2216.72 tok/sec=101624.58 u/sec=25.05	ppl=9.881882 
[2023-12-17:11:35:38:INFO:sockeye.training:__call__] E=4 B=19450	s/sec=2440.98 tok/sec=103770.07 u/sec=25.34	ppl=9.892037 
[2023-12-17:11:35:40:INFO:sockeye.training:__call__] E=4 B=19500	s/sec=2014.21 tok/sec=106624.04 u/sec=23.75	ppl=9.892601 
[2023-12-17:11:35:42:INFO:sockeye.training:__call__] E=4 B=19550	s/sec=1982.70 tok/sec=110847.39 u/sec=23.92	ppl=9.902892 
[2023-12-17:11:35:44:INFO:sockeye.training:__call__] E=4 B=19600	s/sec=2161.43 tok/sec=109054.80 u/sec=24.38	ppl=9.910184 
[2023-12-17:11:35:46:INFO:sockeye.training:__call__] E=4 B=19650	s/sec=2406.30 tok/sec=104727.60 u/sec=25.66	ppl=9.921298 
[2023-12-17:11:35:48:INFO:sockeye.training:__call__] E=4 B=19700	s/sec=2216.72 tok/sec=109331.50 u/sec=24.87	ppl=9.933149 
[2023-12-17:11:35:50:INFO:sockeye.training:__call__] E=4 B=19750	s/sec=2465.94 tok/sec=102569.89 u/sec=25.82	ppl=9.945330 
[2023-12-17:11:35:52:INFO:sockeye.training:__call__] E=4 B=19800	s/sec=2284.72 tok/sec=106319.92 u/sec=25.14	ppl=9.954517 
[2023-12-17:11:35:54:INFO:sockeye.training:__call__] E=4 B=19850	s/sec=2158.32 tok/sec=105474.96 u/sec=25.26	ppl=9.964184 
[2023-12-17:11:35:56:INFO:sockeye.training:__call__] E=4 B=19900	s/sec=2324.00 tok/sec=107286.01 u/sec=24.87	ppl=9.972488 
[2023-12-17:11:35:58:INFO:sockeye.training:__call__] E=4 B=19950	s/sec=2215.14 tok/sec=107321.80 u/sec=25.08	ppl=9.981105 
[2023-12-17:11:36:00:INFO:sockeye.training:__call__] E=5 B=20000	s/sec=1926.05 tok/sec=102612.58 u/sec=23.42	ppl=9.971819 
[2023-12-17:11:36:00:INFO:sockeye.training:_create_checkpoint] Checkpoint [5]	Updates=20000 Epoch=5 Samples=1782008 Time-cost=160.425 Updates/sec=24.934
[2023-12-17:11:36:00:INFO:sockeye.training:_create_checkpoint] Checkpoint [5]	Train-ppl=9.971819
[2023-12-17:11:36:03:INFO:sockeye.training:_evaluate] Checkpoint [5]	Validation-ppl=26.137648	Validation-bleu=0.175208	Validation-chrf=31.983236	Validation-rouge1=0.168988	Validation-rouge2=0.084620	Validation-rougel=0.163078	Validation-length-ratio-mse=1.204812	Validation-ter=92.695006	Validation-avg-sec-per-sent=0.003114	Validation-decode-walltime=1.557038
[2023-12-17:11:36:03:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 31.983236 (delta=1.064246).
[2023-12-17:11:36:03:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [5]	Learning-rate=0.000200
[2023-12-17:11:36:03:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:36:03:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:36:04:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00005"
[2023-12-17:11:36:04:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00005'
[2023-12-17:11:36:04:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:11:36:04:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl'
[2023-12-17:11:36:06:INFO:sockeye.training:__call__] E=5 B=20050	s/sec=692.68 tok/sec=37995.43 u/sec=8.51	ppl=8.312732 
[2023-12-17:11:36:08:INFO:sockeye.training:__call__] E=5 B=20100	s/sec=2389.29 tok/sec=103928.09 u/sec=25.01	ppl=8.364562 
[2023-12-17:11:36:10:INFO:sockeye.training:__call__] E=5 B=20150	s/sec=2356.27 tok/sec=104608.90 u/sec=25.13	ppl=8.402394 
[2023-12-17:11:36:12:INFO:sockeye.training:__call__] E=5 B=20200	s/sec=1927.64 tok/sec=107883.65 u/sec=25.05	ppl=8.390674 
[2023-12-17:11:36:14:INFO:sockeye.training:__call__] E=5 B=20250	s/sec=2147.02 tok/sec=107923.58 u/sec=24.85	ppl=8.405565 
[2023-12-17:11:36:16:INFO:sockeye.training:__call__] E=5 B=20300	s/sec=2277.79 tok/sec=103714.62 u/sec=24.80	ppl=8.451565 
[2023-12-17:11:36:18:INFO:sockeye.training:__call__] E=5 B=20350	s/sec=2225.03 tok/sec=106379.76 u/sec=25.33	ppl=8.474474 
[2023-12-17:11:36:20:INFO:sockeye.training:__call__] E=5 B=20400	s/sec=2289.14 tok/sec=106111.06 u/sec=25.59	ppl=8.485134 
[2023-12-17:11:36:22:INFO:sockeye.training:__call__] E=5 B=20450	s/sec=1969.36 tok/sec=108533.64 u/sec=24.42	ppl=8.472321 
[2023-12-17:11:36:24:INFO:sockeye.training:__call__] E=5 B=20500	s/sec=2156.61 tok/sec=106786.74 u/sec=25.29	ppl=8.493707 
[2023-12-17:11:36:26:INFO:sockeye.training:__call__] E=5 B=20550	s/sec=2184.64 tok/sec=107618.94 u/sec=25.33	ppl=8.546997 
[2023-12-17:11:36:28:INFO:sockeye.training:__call__] E=5 B=20600	s/sec=2259.68 tok/sec=107860.82 u/sec=24.82	ppl=8.568103 
[2023-12-17:11:36:30:INFO:sockeye.training:__call__] E=5 B=20650	s/sec=1972.55 tok/sec=110805.94 u/sec=24.36	ppl=8.580967 
[2023-12-17:11:36:32:INFO:sockeye.training:__call__] E=5 B=20700	s/sec=2168.08 tok/sec=108522.65 u/sec=24.68	ppl=8.606113 
[2023-12-17:11:36:34:INFO:sockeye.training:__call__] E=5 B=20750	s/sec=2365.79 tok/sec=106335.59 u/sec=25.19	ppl=8.632571 
[2023-12-17:11:36:36:INFO:sockeye.training:__call__] E=5 B=20800	s/sec=2321.75 tok/sec=103943.03 u/sec=25.37	ppl=8.641954 
[2023-12-17:11:36:38:INFO:sockeye.training:__call__] E=5 B=20850	s/sec=2267.36 tok/sec=107193.01 u/sec=24.65	ppl=8.656347 
[2023-12-17:11:36:40:INFO:sockeye.training:__call__] E=5 B=20900	s/sec=2186.51 tok/sec=105804.19 u/sec=23.77	ppl=8.676350 
[2023-12-17:11:36:42:INFO:sockeye.training:__call__] E=5 B=20950	s/sec=2174.18 tok/sec=105698.43 u/sec=24.89	ppl=8.686694 
[2023-12-17:11:36:44:INFO:sockeye.training:__call__] E=5 B=21000	s/sec=2192.09 tok/sec=108290.87 u/sec=24.73	ppl=8.705998 
[2023-12-17:11:36:46:INFO:sockeye.training:__call__] E=5 B=21050	s/sec=2418.15 tok/sec=102075.97 u/sec=25.49	ppl=8.728302 
[2023-12-17:11:36:48:INFO:sockeye.training:__call__] E=5 B=21100	s/sec=2008.18 tok/sec=102323.24 u/sec=24.51	ppl=8.736501 
[2023-12-17:11:36:50:INFO:sockeye.training:__call__] E=5 B=21150	s/sec=2171.88 tok/sec=102631.95 u/sec=23.86	ppl=8.743320 
[2023-12-17:11:36:52:INFO:sockeye.training:__call__] E=5 B=21200	s/sec=2316.54 tok/sec=105027.32 u/sec=25.76	ppl=8.760357 
[2023-12-17:11:36:54:INFO:sockeye.training:__call__] E=5 B=21250	s/sec=2493.44 tok/sec=100978.01 u/sec=26.19	ppl=8.775830 
[2023-12-17:11:36:56:INFO:sockeye.training:__call__] E=5 B=21300	s/sec=2258.62 tok/sec=107548.72 u/sec=25.03	ppl=8.783487 
[2023-12-17:11:36:58:INFO:sockeye.training:__call__] E=5 B=21350	s/sec=2410.73 tok/sec=103730.77 u/sec=25.58	ppl=8.794880 
[2023-12-17:11:37:00:INFO:sockeye.training:__call__] E=5 B=21400	s/sec=2240.77 tok/sec=109675.28 u/sec=24.70	ppl=8.811026 
[2023-12-17:11:37:02:INFO:sockeye.training:__call__] E=5 B=21450	s/sec=2240.18 tok/sec=107307.20 u/sec=24.69	ppl=8.824652 
[2023-12-17:11:37:04:INFO:sockeye.training:__call__] E=5 B=21500	s/sec=2619.40 tok/sec=100545.29 u/sec=25.82	ppl=8.846253 
[2023-12-17:11:37:06:INFO:sockeye.training:__call__] E=5 B=21550	s/sec=2204.88 tok/sec=100635.32 u/sec=24.18	ppl=8.862228 
[2023-12-17:11:37:08:INFO:sockeye.training:__call__] E=5 B=21600	s/sec=2504.01 tok/sec=95265.14 u/sec=25.16	ppl=8.883150 
[2023-12-17:11:37:10:INFO:sockeye.training:__call__] E=5 B=21650	s/sec=2078.43 tok/sec=111029.23 u/sec=24.79	ppl=8.891582 
[2023-12-17:11:37:12:INFO:sockeye.training:__call__] E=5 B=21700	s/sec=2217.76 tok/sec=108567.47 u/sec=24.66	ppl=8.906115 
[2023-12-17:11:37:14:INFO:sockeye.training:__call__] E=5 B=21750	s/sec=2256.01 tok/sec=113718.05 u/sec=23.31	ppl=8.912837 
[2023-12-17:11:37:16:INFO:sockeye.training:__call__] E=5 B=21800	s/sec=2355.14 tok/sec=105050.71 u/sec=25.29	ppl=8.924348 
[2023-12-17:11:37:18:INFO:sockeye.training:__call__] E=5 B=21850	s/sec=2433.65 tok/sec=103748.86 u/sec=25.39	ppl=8.938218 
[2023-12-17:11:37:20:INFO:sockeye.training:__call__] E=5 B=21900	s/sec=2092.80 tok/sec=104592.11 u/sec=24.87	ppl=8.952455 
[2023-12-17:11:37:22:INFO:sockeye.training:__call__] E=5 B=21950	s/sec=2040.32 tok/sec=111068.69 u/sec=24.43	ppl=8.958692 
[2023-12-17:11:37:24:INFO:sockeye.training:__call__] E=5 B=22000	s/sec=2166.82 tok/sec=105997.70 u/sec=25.08	ppl=8.971909 
[2023-12-17:11:37:27:INFO:sockeye.training:__call__] E=5 B=22050	s/sec=1938.78 tok/sec=108789.97 u/sec=24.33	ppl=8.977182 
[2023-12-17:11:37:29:INFO:sockeye.training:__call__] E=5 B=22100	s/sec=2038.51 tok/sec=110103.30 u/sec=24.93	ppl=8.984316 
[2023-12-17:11:37:31:INFO:sockeye.training:__call__] E=5 B=22150	s/sec=2120.40 tok/sec=107756.63 u/sec=25.24	ppl=8.994552 
[2023-12-17:11:37:33:INFO:sockeye.training:__call__] E=5 B=22200	s/sec=2262.75 tok/sec=105819.62 u/sec=25.07	ppl=9.007369 
[2023-12-17:11:37:34:INFO:sockeye.training:__call__] E=5 B=22250	s/sec=2289.04 tok/sec=106270.12 u/sec=25.37	ppl=9.019491 
[2023-12-17:11:37:36:INFO:sockeye.training:__call__] E=5 B=22300	s/sec=2230.77 tok/sec=106328.01 u/sec=25.44	ppl=9.028051 
[2023-12-17:11:37:38:INFO:sockeye.training:__call__] E=5 B=22350	s/sec=2217.77 tok/sec=105634.69 u/sec=25.57	ppl=9.040839 
[2023-12-17:11:37:40:INFO:sockeye.training:__call__] E=5 B=22400	s/sec=2482.71 tok/sec=103761.75 u/sec=25.40	ppl=9.054281 
[2023-12-17:11:37:42:INFO:sockeye.training:__call__] E=5 B=22450	s/sec=2480.24 tok/sec=100650.17 u/sec=25.58	ppl=9.069582 
[2023-12-17:11:37:44:INFO:sockeye.training:__call__] E=5 B=22500	s/sec=2144.12 tok/sec=106602.30 u/sec=24.82	ppl=9.082851 
[2023-12-17:11:37:46:INFO:sockeye.training:__call__] E=5 B=22550	s/sec=2259.99 tok/sec=106611.46 u/sec=24.74	ppl=9.093250 
[2023-12-17:11:37:48:INFO:sockeye.training:__call__] E=5 B=22600	s/sec=2438.50 tok/sec=105284.34 u/sec=25.53	ppl=9.108961 
[2023-12-17:11:37:50:INFO:sockeye.training:__call__] E=5 B=22650	s/sec=2590.35 tok/sec=103219.86 u/sec=25.66	ppl=9.121234 
[2023-12-17:11:37:52:INFO:sockeye.training:__call__] E=5 B=22700	s/sec=2114.71 tok/sec=108398.28 u/sec=24.48	ppl=9.127049 
[2023-12-17:11:37:54:INFO:sockeye.training:__call__] E=5 B=22750	s/sec=2183.61 tok/sec=107946.52 u/sec=24.72	ppl=9.138342 
[2023-12-17:11:37:56:INFO:sockeye.training:__call__] E=5 B=22800	s/sec=2079.07 tok/sec=110182.81 u/sec=24.89	ppl=9.143373 
[2023-12-17:11:37:58:INFO:sockeye.training:__call__] E=5 B=22850	s/sec=2017.33 tok/sec=108729.90 u/sec=24.72	ppl=9.153357 
[2023-12-17:11:38:00:INFO:sockeye.training:__call__] E=5 B=22900	s/sec=2001.94 tok/sec=105405.29 u/sec=23.70	ppl=9.162917 
[2023-12-17:11:38:02:INFO:sockeye.training:__call__] E=5 B=22950	s/sec=2322.05 tok/sec=107736.83 u/sec=25.28	ppl=9.171681 
[2023-12-17:11:38:04:INFO:sockeye.training:__call__] E=5 B=23000	s/sec=2464.79 tok/sec=105087.44 u/sec=25.76	ppl=9.187783 
[2023-12-17:11:38:06:INFO:sockeye.training:__call__] E=5 B=23050	s/sec=2452.78 tok/sec=104824.24 u/sec=25.21	ppl=9.197349 
[2023-12-17:11:38:08:INFO:sockeye.training:__call__] E=5 B=23100	s/sec=2422.48 tok/sec=105652.55 u/sec=25.23	ppl=9.208992 
[2023-12-17:11:38:10:INFO:sockeye.training:__call__] E=5 B=23150	s/sec=2074.42 tok/sec=112991.72 u/sec=24.32	ppl=9.214517 
[2023-12-17:11:38:12:INFO:sockeye.training:__call__] E=5 B=23200	s/sec=2110.53 tok/sec=108040.41 u/sec=24.47	ppl=9.224298 
[2023-12-17:11:38:14:INFO:sockeye.training:__call__] E=5 B=23250	s/sec=2349.66 tok/sec=106873.06 u/sec=25.23	ppl=9.234555 
[2023-12-17:11:38:16:INFO:sockeye.training:__call__] E=5 B=23300	s/sec=2106.73 tok/sec=107979.86 u/sec=25.03	ppl=9.240308 
[2023-12-17:11:38:19:INFO:sockeye.training:__call__] E=5 B=23350	s/sec=2064.57 tok/sec=108985.02 u/sec=23.76	ppl=9.252122 
[2023-12-17:11:38:21:INFO:sockeye.training:__call__] E=5 B=23400	s/sec=2102.04 tok/sec=103484.88 u/sec=24.06	ppl=9.259168 
[2023-12-17:11:38:23:INFO:sockeye.training:__call__] E=5 B=23450	s/sec=2199.16 tok/sec=109778.22 u/sec=24.41	ppl=9.268920 
[2023-12-17:11:38:25:INFO:sockeye.training:__call__] E=5 B=23500	s/sec=2003.15 tok/sec=105530.14 u/sec=24.26	ppl=9.274381 
[2023-12-17:11:38:27:INFO:sockeye.training:__call__] E=5 B=23550	s/sec=2311.77 tok/sec=106672.17 u/sec=25.22	ppl=9.283502 
[2023-12-17:11:38:29:INFO:sockeye.training:__call__] E=5 B=23600	s/sec=2318.70 tok/sec=105915.13 u/sec=25.03	ppl=9.293539 
[2023-12-17:11:38:31:INFO:sockeye.training:__call__] E=5 B=23650	s/sec=2149.05 tok/sec=109112.86 u/sec=24.24	ppl=9.302120 
[2023-12-17:11:38:33:INFO:sockeye.training:__call__] E=5 B=23700	s/sec=2275.59 tok/sec=107288.30 u/sec=25.26	ppl=9.310158 
[2023-12-17:11:38:35:INFO:sockeye.training:__call__] E=5 B=23750	s/sec=2202.79 tok/sec=106553.01 u/sec=25.59	ppl=9.317228 
[2023-12-17:11:38:37:INFO:sockeye.training:__call__] E=5 B=23800	s/sec=2222.14 tok/sec=101661.02 u/sec=24.89	ppl=9.321441 
[2023-12-17:11:38:39:INFO:sockeye.training:__call__] E=5 B=23850	s/sec=2154.59 tok/sec=108062.35 u/sec=24.18	ppl=9.329082 
[2023-12-17:11:38:41:INFO:sockeye.training:__call__] E=5 B=23900	s/sec=1968.41 tok/sec=112029.79 u/sec=24.65	ppl=9.333308 
[2023-12-17:11:38:43:INFO:sockeye.training:__call__] E=5 B=23950	s/sec=2191.88 tok/sec=105427.15 u/sec=24.20	ppl=9.345251 
[2023-12-17:11:38:45:INFO:sockeye.training:__call__] E=6 B=24000	s/sec=1858.21 tok/sec=98942.76 u/sec=22.33	ppl=9.335833 
[2023-12-17:11:38:45:INFO:sockeye.training:_create_checkpoint] Checkpoint [6]	Updates=24000 Epoch=6 Samples=2138832 Time-cost=160.980 Updates/sec=24.848
[2023-12-17:11:38:45:INFO:sockeye.training:_create_checkpoint] Checkpoint [6]	Train-ppl=9.335833
[2023-12-17:11:38:48:INFO:sockeye.training:_evaluate] Checkpoint [6]	Validation-ppl=27.926656	Validation-bleu=0.139581	Validation-chrf=29.959064	Validation-rouge1=0.159920	Validation-rouge2=0.072936	Validation-rougel=0.152293	Validation-length-ratio-mse=1.290122	Validation-ter=97.730087	Validation-avg-sec-per-sent=0.003217	Validation-decode-walltime=1.608505
[2023-12-17:11:38:48:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 1 checkpoints, best so far: 31.983236
[2023-12-17:11:38:48:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [6]	Learning-rate=0.000200
[2023-12-17:11:38:48:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:38:48:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:38:48:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00006"
[2023-12-17:11:38:51:INFO:sockeye.training:__call__] E=6 B=24050	s/sec=784.72 tok/sec=40270.44 u/sec=9.29	ppl=7.673105 
[2023-12-17:11:38:53:INFO:sockeye.training:__call__] E=6 B=24100	s/sec=2055.91 tok/sec=109130.34 u/sec=24.29	ppl=7.784262 
[2023-12-17:11:38:55:INFO:sockeye.training:__call__] E=6 B=24150	s/sec=2162.08 tok/sec=107251.93 u/sec=25.35	ppl=7.812009 
[2023-12-17:11:38:57:INFO:sockeye.training:__call__] E=6 B=24200	s/sec=2320.61 tok/sec=105178.88 u/sec=25.67	ppl=7.860158 
[2023-12-17:11:38:58:INFO:sockeye.training:__call__] E=6 B=24250	s/sec=2254.86 tok/sec=106710.96 u/sec=25.30	ppl=7.895305 
[2023-12-17:11:39:01:INFO:sockeye.training:__call__] E=6 B=24300	s/sec=2183.41 tok/sec=110175.99 u/sec=24.54	ppl=7.891324 
[2023-12-17:11:39:03:INFO:sockeye.training:__call__] E=6 B=24350	s/sec=2056.01 tok/sec=102717.67 u/sec=23.49	ppl=7.916561 
[2023-12-17:11:39:05:INFO:sockeye.training:__call__] E=6 B=24400	s/sec=1938.58 tok/sec=110644.39 u/sec=23.94	ppl=7.929631 
[2023-12-17:11:39:07:INFO:sockeye.training:__call__] E=6 B=24450	s/sec=2046.00 tok/sec=104064.57 u/sec=23.46	ppl=7.943886 
[2023-12-17:11:39:09:INFO:sockeye.training:__call__] E=6 B=24500	s/sec=2285.32 tok/sec=107180.48 u/sec=25.19	ppl=7.967228 
[2023-12-17:11:39:11:INFO:sockeye.training:__call__] E=6 B=24550	s/sec=2044.22 tok/sec=111777.23 u/sec=24.81	ppl=7.974400 
[2023-12-17:11:39:13:INFO:sockeye.training:__call__] E=6 B=24600	s/sec=2315.87 tok/sec=114254.58 u/sec=23.69	ppl=8.001292 
[2023-12-17:11:39:15:INFO:sockeye.training:__call__] E=6 B=24650	s/sec=2062.84 tok/sec=108026.62 u/sec=24.70	ppl=7.997088 
[2023-12-17:11:39:17:INFO:sockeye.training:__call__] E=6 B=24700	s/sec=2304.67 tok/sec=104908.91 u/sec=25.23	ppl=8.017593 
[2023-12-17:11:39:19:INFO:sockeye.training:__call__] E=6 B=24750	s/sec=2129.88 tok/sec=105603.35 u/sec=25.31	ppl=8.019784 
[2023-12-17:11:39:21:INFO:sockeye.training:__call__] E=6 B=24800	s/sec=2396.46 tok/sec=105542.64 u/sec=25.65	ppl=8.038520 
[2023-12-17:11:39:23:INFO:sockeye.training:__call__] E=6 B=24850	s/sec=2309.01 tok/sec=103656.27 u/sec=25.96	ppl=8.060672 
[2023-12-17:11:39:25:INFO:sockeye.training:__call__] E=6 B=24900	s/sec=2440.94 tok/sec=105378.12 u/sec=25.09	ppl=8.090107 
[2023-12-17:11:39:27:INFO:sockeye.training:__call__] E=6 B=24950	s/sec=2269.86 tok/sec=110438.39 u/sec=24.17	ppl=8.106067 
[2023-12-17:11:39:29:INFO:sockeye.training:__call__] E=6 B=25000	s/sec=2211.45 tok/sec=110360.44 u/sec=24.55	ppl=8.127705 
[2023-12-17:11:39:31:INFO:sockeye.training:__call__] E=6 B=25050	s/sec=2204.47 tok/sec=108599.20 u/sec=25.01	ppl=8.144946 
[2023-12-17:11:39:33:INFO:sockeye.training:__call__] E=6 B=25100	s/sec=2267.91 tok/sec=105185.70 u/sec=25.40	ppl=8.164599 
[2023-12-17:11:39:35:INFO:sockeye.training:__call__] E=6 B=25150	s/sec=2554.67 tok/sec=104457.57 u/sec=25.71	ppl=8.187195 
[2023-12-17:11:39:37:INFO:sockeye.training:__call__] E=6 B=25200	s/sec=2247.17 tok/sec=106368.84 u/sec=24.86	ppl=8.208164 
[2023-12-17:11:39:39:INFO:sockeye.training:__call__] E=6 B=25250	s/sec=2089.13 tok/sec=110142.57 u/sec=24.54	ppl=8.223114 
[2023-12-17:11:39:41:INFO:sockeye.training:__call__] E=6 B=25300	s/sec=2208.82 tok/sec=106435.76 u/sec=24.78	ppl=8.234559 
[2023-12-17:11:39:43:INFO:sockeye.training:__call__] E=6 B=25350	s/sec=2097.08 tok/sec=108119.50 u/sec=25.01	ppl=8.246356 
[2023-12-17:11:39:45:INFO:sockeye.training:__call__] E=6 B=25400	s/sec=2412.70 tok/sec=104248.38 u/sec=25.52	ppl=8.260521 
[2023-12-17:11:39:47:INFO:sockeye.training:__call__] E=6 B=25450	s/sec=1968.78 tok/sec=112122.98 u/sec=24.37	ppl=8.264884 
[2023-12-17:11:39:49:INFO:sockeye.training:__call__] E=6 B=25500	s/sec=2229.03 tok/sec=107470.86 u/sec=24.88	ppl=8.280757 
[2023-12-17:11:39:51:INFO:sockeye.training:__call__] E=6 B=25550	s/sec=2561.16 tok/sec=102577.88 u/sec=25.69	ppl=8.300114 
[2023-12-17:11:39:53:INFO:sockeye.training:__call__] E=6 B=25600	s/sec=2322.85 tok/sec=102658.79 u/sec=25.29	ppl=8.313323 
[2023-12-17:11:39:55:INFO:sockeye.training:__call__] E=6 B=25650	s/sec=1994.53 tok/sec=103925.71 u/sec=23.43	ppl=8.317091 
[2023-12-17:11:39:57:INFO:sockeye.training:__call__] E=6 B=25700	s/sec=2392.67 tok/sec=103924.15 u/sec=25.48	ppl=8.331420 
[2023-12-17:11:39:59:INFO:sockeye.training:__call__] E=6 B=25750	s/sec=2286.64 tok/sec=106488.58 u/sec=25.29	ppl=8.348136 
[2023-12-17:11:40:01:INFO:sockeye.training:__call__] E=6 B=25800	s/sec=1955.66 tok/sec=112498.29 u/sec=24.01	ppl=8.353168 
[2023-12-17:11:40:03:INFO:sockeye.training:__call__] E=6 B=25850	s/sec=2242.29 tok/sec=107341.78 u/sec=25.03	ppl=8.368899 
[2023-12-17:11:40:05:INFO:sockeye.training:__call__] E=6 B=25900	s/sec=2114.32 tok/sec=106219.74 u/sec=24.98	ppl=8.380126 
[2023-12-17:11:40:07:INFO:sockeye.training:__call__] E=6 B=25950	s/sec=1855.60 tok/sec=110948.48 u/sec=24.21	ppl=8.388803 
[2023-12-17:11:40:09:INFO:sockeye.training:__call__] E=6 B=26000	s/sec=2548.18 tok/sec=101662.26 u/sec=25.85	ppl=8.405251 
[2023-12-17:11:40:11:INFO:sockeye.training:__call__] E=6 B=26050	s/sec=2094.36 tok/sec=104583.16 u/sec=24.74	ppl=8.419381 
[2023-12-17:11:40:13:INFO:sockeye.training:__call__] E=6 B=26100	s/sec=2285.86 tok/sec=100453.26 u/sec=24.30	ppl=8.431631 
[2023-12-17:11:40:15:INFO:sockeye.training:__call__] E=6 B=26150	s/sec=2201.22 tok/sec=105106.36 u/sec=23.97	ppl=8.447868 
[2023-12-17:11:40:17:INFO:sockeye.training:__call__] E=6 B=26200	s/sec=2046.49 tok/sec=109042.15 u/sec=24.41	ppl=8.457952 
[2023-12-17:11:40:19:INFO:sockeye.training:__call__] E=6 B=26250	s/sec=2405.14 tok/sec=106004.19 u/sec=25.26	ppl=8.473095 
[2023-12-17:11:40:21:INFO:sockeye.training:__call__] E=6 B=26300	s/sec=2250.39 tok/sec=105483.23 u/sec=24.76	ppl=8.483119 
[2023-12-17:11:40:23:INFO:sockeye.training:__call__] E=6 B=26350	s/sec=2122.39 tok/sec=109572.14 u/sec=24.75	ppl=8.492951 
[2023-12-17:11:40:25:INFO:sockeye.training:__call__] E=6 B=26400	s/sec=2216.60 tok/sec=109426.66 u/sec=24.92	ppl=8.503447 
[2023-12-17:11:40:27:INFO:sockeye.training:__call__] E=6 B=26450	s/sec=2032.77 tok/sec=110102.03 u/sec=24.72	ppl=8.515199 
[2023-12-17:11:40:29:INFO:sockeye.training:__call__] E=6 B=26500	s/sec=2255.74 tok/sec=106362.68 u/sec=24.91	ppl=8.524718 
[2023-12-17:11:40:31:INFO:sockeye.training:__call__] E=6 B=26550	s/sec=2101.10 tok/sec=104566.19 u/sec=25.45	ppl=8.534704 
[2023-12-17:11:40:33:INFO:sockeye.training:__call__] E=6 B=26600	s/sec=2212.05 tok/sec=108311.11 u/sec=24.87	ppl=8.547652 
[2023-12-17:11:40:35:INFO:sockeye.training:__call__] E=6 B=26650	s/sec=2127.60 tok/sec=105381.86 u/sec=24.95	ppl=8.554857 
[2023-12-17:11:40:37:INFO:sockeye.training:__call__] E=6 B=26700	s/sec=2290.11 tok/sec=104976.28 u/sec=25.60	ppl=8.569227 
[2023-12-17:11:40:39:INFO:sockeye.training:__call__] E=6 B=26750	s/sec=2536.01 tok/sec=98588.22 u/sec=25.32	ppl=8.582705 
[2023-12-17:11:40:41:INFO:sockeye.training:__call__] E=6 B=26800	s/sec=2186.35 tok/sec=103746.00 u/sec=24.02	ppl=8.594748 
[2023-12-17:11:40:43:INFO:sockeye.training:__call__] E=6 B=26850	s/sec=2444.83 tok/sec=102190.62 u/sec=25.64	ppl=8.608649 
[2023-12-17:11:40:45:INFO:sockeye.training:__call__] E=6 B=26900	s/sec=2216.67 tok/sec=106976.54 u/sec=25.01	ppl=8.620716 
[2023-12-17:11:40:47:INFO:sockeye.training:__call__] E=6 B=26950	s/sec=2468.92 tok/sec=105752.79 u/sec=25.42	ppl=8.632968 
[2023-12-17:11:40:49:INFO:sockeye.training:__call__] E=6 B=27000	s/sec=2131.44 tok/sec=108679.64 u/sec=24.76	ppl=8.645294 
[2023-12-17:11:40:51:INFO:sockeye.training:__call__] E=6 B=27050	s/sec=2196.28 tok/sec=106743.20 u/sec=25.19	ppl=8.656513 
[2023-12-17:11:40:53:INFO:sockeye.training:__call__] E=6 B=27100	s/sec=2192.84 tok/sec=106196.84 u/sec=25.33	ppl=8.665638 
[2023-12-17:11:40:55:INFO:sockeye.training:__call__] E=6 B=27150	s/sec=2181.22 tok/sec=108519.48 u/sec=24.88	ppl=8.675521 
[2023-12-17:11:40:57:INFO:sockeye.training:__call__] E=6 B=27200	s/sec=2328.91 tok/sec=102714.14 u/sec=25.23	ppl=8.685204 
[2023-12-17:11:40:59:INFO:sockeye.training:__call__] E=6 B=27250	s/sec=2370.29 tok/sec=106433.64 u/sec=25.15	ppl=8.696401 
[2023-12-17:11:41:01:INFO:sockeye.training:__call__] E=6 B=27300	s/sec=2159.96 tok/sec=108111.20 u/sec=25.28	ppl=8.706694 
[2023-12-17:11:41:03:INFO:sockeye.training:__call__] E=6 B=27350	s/sec=2176.31 tok/sec=108591.62 u/sec=25.00	ppl=8.715025 
[2023-12-17:11:41:05:INFO:sockeye.training:__call__] E=6 B=27400	s/sec=2274.45 tok/sec=105229.00 u/sec=24.85	ppl=8.723289 
[2023-12-17:11:41:07:INFO:sockeye.training:__call__] E=6 B=27450	s/sec=1770.27 tok/sec=114264.11 u/sec=24.37	ppl=8.729853 
[2023-12-17:11:41:09:INFO:sockeye.training:__call__] E=6 B=27500	s/sec=2316.60 tok/sec=105193.39 u/sec=25.49	ppl=8.741081 
[2023-12-17:11:41:11:INFO:sockeye.training:__call__] E=6 B=27550	s/sec=2561.73 tok/sec=101096.36 u/sec=26.16	ppl=8.752131 
[2023-12-17:11:41:13:INFO:sockeye.training:__call__] E=6 B=27600	s/sec=2487.98 tok/sec=105586.46 u/sec=25.08	ppl=8.764073 
[2023-12-17:11:41:15:INFO:sockeye.training:__call__] E=6 B=27650	s/sec=2320.50 tok/sec=105715.98 u/sec=24.88	ppl=8.770549 
[2023-12-17:11:41:17:INFO:sockeye.training:__call__] E=6 B=27700	s/sec=2048.75 tok/sec=110899.13 u/sec=24.53	ppl=8.779267 
[2023-12-17:11:41:19:INFO:sockeye.training:__call__] E=6 B=27750	s/sec=2163.19 tok/sec=107404.48 u/sec=25.37	ppl=8.786765 
[2023-12-17:11:41:21:INFO:sockeye.training:__call__] E=6 B=27800	s/sec=2206.30 tok/sec=106224.62 u/sec=25.16	ppl=8.793562 
[2023-12-17:11:41:23:INFO:sockeye.training:__call__] E=6 B=27850	s/sec=2340.79 tok/sec=103557.36 u/sec=25.85	ppl=8.803753 
[2023-12-17:11:41:25:INFO:sockeye.training:__call__] E=6 B=27900	s/sec=2161.37 tok/sec=108262.03 u/sec=25.20	ppl=8.811251 
[2023-12-17:11:41:27:INFO:sockeye.training:__call__] E=6 B=27950	s/sec=2312.03 tok/sec=107631.86 u/sec=25.13	ppl=8.818142 
[2023-12-17:11:41:29:INFO:sockeye.training:__call__] E=7 B=28000	s/sec=2181.77 tok/sec=101288.58 u/sec=23.55	ppl=8.809211 
[2023-12-17:11:41:29:INFO:sockeye.training:_create_checkpoint] Checkpoint [7]	Updates=28000 Epoch=7 Samples=2495344 Time-cost=160.694 Updates/sec=24.892
[2023-12-17:11:41:29:INFO:sockeye.training:_create_checkpoint] Checkpoint [7]	Train-ppl=8.809211
[2023-12-17:11:41:32:INFO:sockeye.training:_evaluate] Checkpoint [7]	Validation-ppl=28.490051	Validation-bleu=0.154481	Validation-chrf=30.240848	Validation-rouge1=0.154599	Validation-rouge2=0.076176	Validation-rougel=0.146774	Validation-length-ratio-mse=1.323126	Validation-ter=98.803137	Validation-avg-sec-per-sent=0.003546	Validation-decode-walltime=1.773142
[2023-12-17:11:41:32:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 2 checkpoints, best so far: 31.983236
[2023-12-17:11:41:32:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [7]	Learning-rate=0.000200
[2023-12-17:11:41:32:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:41:32:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:41:33:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00007"
[2023-12-17:11:41:35:INFO:sockeye.training:__call__] E=7 B=28050	s/sec=778.51 tok/sec=36840.06 u/sec=8.98	ppl=7.662700 
[2023-12-17:11:41:37:INFO:sockeye.training:__call__] E=7 B=28100	s/sec=2211.97 tok/sec=106780.36 u/sec=24.91	ppl=7.550324 
[2023-12-17:11:41:39:INFO:sockeye.training:__call__] E=7 B=28150	s/sec=2128.58 tok/sec=110917.96 u/sec=24.14	ppl=7.537176 
[2023-12-17:11:41:41:INFO:sockeye.training:__call__] E=7 B=28200	s/sec=2455.10 tok/sec=107214.13 u/sec=25.32	ppl=7.524609 
[2023-12-17:11:41:43:INFO:sockeye.training:__call__] E=7 B=28250	s/sec=2010.43 tok/sec=109608.11 u/sec=24.73	ppl=7.527439 
[2023-12-17:11:41:45:INFO:sockeye.training:__call__] E=7 B=28300	s/sec=2077.00 tok/sec=107672.87 u/sec=24.63	ppl=7.534394 
[2023-12-17:11:41:47:INFO:sockeye.training:__call__] E=7 B=28350	s/sec=2108.98 tok/sec=105676.06 u/sec=25.35	ppl=7.563868 
[2023-12-17:11:41:49:INFO:sockeye.training:__call__] E=7 B=28400	s/sec=2175.56 tok/sec=106678.84 u/sec=25.13	ppl=7.562246 
[2023-12-17:11:41:51:INFO:sockeye.training:__call__] E=7 B=28450	s/sec=2221.51 tok/sec=106242.68 u/sec=25.38	ppl=7.573980 
[2023-12-17:11:41:53:INFO:sockeye.training:__call__] E=7 B=28500	s/sec=2106.90 tok/sec=96770.58 u/sec=24.16	ppl=7.598141 
[2023-12-17:11:41:55:INFO:sockeye.training:__call__] E=7 B=28550	s/sec=2099.75 tok/sec=111523.45 u/sec=24.67	ppl=7.616298 
[2023-12-17:11:41:57:INFO:sockeye.training:__call__] E=7 B=28600	s/sec=1970.55 tok/sec=108994.90 u/sec=24.63	ppl=7.622916 
[2023-12-17:11:41:59:INFO:sockeye.training:__call__] E=7 B=28650	s/sec=2170.27 tok/sec=109251.36 u/sec=24.80	ppl=7.640320 
[2023-12-17:11:42:01:INFO:sockeye.training:__call__] E=7 B=28700	s/sec=2092.16 tok/sec=111229.89 u/sec=24.26	ppl=7.654318 
[2023-12-17:11:42:03:INFO:sockeye.training:__call__] E=7 B=28750	s/sec=2339.26 tok/sec=105054.07 u/sec=25.56	ppl=7.669898 
[2023-12-17:11:42:05:INFO:sockeye.training:__call__] E=7 B=28800	s/sec=2145.56 tok/sec=108643.99 u/sec=25.11	ppl=7.680952 
[2023-12-17:11:42:07:INFO:sockeye.training:__call__] E=7 B=28850	s/sec=2357.91 tok/sec=103308.24 u/sec=25.45	ppl=7.701741 
[2023-12-17:11:42:09:INFO:sockeye.training:__call__] E=7 B=28900	s/sec=2160.07 tok/sec=109190.41 u/sec=24.73	ppl=7.720147 
[2023-12-17:11:42:11:INFO:sockeye.training:__call__] E=7 B=28950	s/sec=2235.55 tok/sec=107828.78 u/sec=24.01	ppl=7.743600 
[2023-12-17:11:42:13:INFO:sockeye.training:__call__] E=7 B=29000	s/sec=2268.12 tok/sec=101089.23 u/sec=23.75	ppl=7.769144 
[2023-12-17:11:42:15:INFO:sockeye.training:__call__] E=7 B=29050	s/sec=2340.86 tok/sec=100390.54 u/sec=25.62	ppl=7.781144 
[2023-12-17:11:42:17:INFO:sockeye.training:__call__] E=7 B=29100	s/sec=2375.89 tok/sec=105887.40 u/sec=25.38	ppl=7.799289 
[2023-12-17:11:42:19:INFO:sockeye.training:__call__] E=7 B=29150	s/sec=2301.85 tok/sec=105747.84 u/sec=25.20	ppl=7.812795 
[2023-12-17:11:42:21:INFO:sockeye.training:__call__] E=7 B=29200	s/sec=2049.07 tok/sec=109889.97 u/sec=24.49	ppl=7.824678 
[2023-12-17:11:42:23:INFO:sockeye.training:__call__] E=7 B=29250	s/sec=2506.37 tok/sec=101820.06 u/sec=26.02	ppl=7.843846 
[2023-12-17:11:42:25:INFO:sockeye.training:__call__] E=7 B=29300	s/sec=2181.63 tok/sec=106260.83 u/sec=25.11	ppl=7.855016 
[2023-12-17:11:42:27:INFO:sockeye.training:__call__] E=7 B=29350	s/sec=2324.33 tok/sec=104396.19 u/sec=25.35	ppl=7.868946 
[2023-12-17:11:42:29:INFO:sockeye.training:__call__] E=7 B=29400	s/sec=2090.72 tok/sec=107604.62 u/sec=24.84	ppl=7.878014 
[2023-12-17:11:42:31:INFO:sockeye.training:__call__] E=7 B=29450	s/sec=2024.98 tok/sec=111744.74 u/sec=24.39	ppl=7.887563 
[2023-12-17:11:42:33:INFO:sockeye.training:__call__] E=7 B=29500	s/sec=2190.35 tok/sec=109051.88 u/sec=24.67	ppl=7.902502 
[2023-12-17:11:42:35:INFO:sockeye.training:__call__] E=7 B=29550	s/sec=2071.98 tok/sec=112308.96 u/sec=24.43	ppl=7.911449 
[2023-12-17:11:42:37:INFO:sockeye.training:__call__] E=7 B=29600	s/sec=2028.51 tok/sec=111436.85 u/sec=24.10	ppl=7.922085 
[2023-12-17:11:42:39:INFO:sockeye.training:__call__] E=7 B=29650	s/sec=2523.58 tok/sec=104411.58 u/sec=25.56	ppl=7.939225 
[2023-12-17:11:42:41:INFO:sockeye.training:__call__] E=7 B=29700	s/sec=2414.14 tok/sec=106925.77 u/sec=24.98	ppl=7.950084 
[2023-12-17:11:42:43:INFO:sockeye.training:__call__] E=7 B=29750	s/sec=2190.88 tok/sec=107723.34 u/sec=24.11	ppl=7.961450 
[2023-12-17:11:42:45:INFO:sockeye.training:__call__] E=7 B=29800	s/sec=2199.45 tok/sec=106733.87 u/sec=25.18	ppl=7.972477 
[2023-12-17:11:42:47:INFO:sockeye.training:__call__] E=7 B=29850	s/sec=2228.60 tok/sec=107166.34 u/sec=25.19	ppl=7.986896 
[2023-12-17:11:42:49:INFO:sockeye.training:__call__] E=7 B=29900	s/sec=2077.33 tok/sec=108782.49 u/sec=24.78	ppl=7.996374 
[2023-12-17:11:42:51:INFO:sockeye.training:__call__] E=7 B=29950	s/sec=2326.08 tok/sec=101972.65 u/sec=25.46	ppl=8.013181 
[2023-12-17:11:42:53:INFO:sockeye.training:__call__] E=7 B=30000	s/sec=2156.70 tok/sec=104831.11 u/sec=25.58	ppl=8.023223 
[2023-12-17:11:42:55:INFO:sockeye.training:__call__] E=7 B=30050	s/sec=2141.83 tok/sec=107067.50 u/sec=24.84	ppl=8.033999 
[2023-12-17:11:42:57:INFO:sockeye.training:__call__] E=7 B=30100	s/sec=2158.47 tok/sec=107180.21 u/sec=24.71	ppl=8.044805 
[2023-12-17:11:42:59:INFO:sockeye.training:__call__] E=7 B=30150	s/sec=2043.36 tok/sec=108978.95 u/sec=23.52	ppl=8.055748 
[2023-12-17:11:43:01:INFO:sockeye.training:__call__] E=7 B=30200	s/sec=2242.99 tok/sec=101286.19 u/sec=24.42	ppl=8.066602 
[2023-12-17:11:43:03:INFO:sockeye.training:__call__] E=7 B=30250	s/sec=2399.74 tok/sec=104160.95 u/sec=25.64	ppl=8.079969 
[2023-12-17:11:43:05:INFO:sockeye.training:__call__] E=7 B=30300	s/sec=2331.14 tok/sec=104979.36 u/sec=25.61	ppl=8.089466 
[2023-12-17:11:43:07:INFO:sockeye.training:__call__] E=7 B=30350	s/sec=2461.79 tok/sec=104971.73 u/sec=25.26	ppl=8.101776 
[2023-12-17:11:43:09:INFO:sockeye.training:__call__] E=7 B=30400	s/sec=2498.72 tok/sec=102900.31 u/sec=26.20	ppl=8.114589 
[2023-12-17:11:43:11:INFO:sockeye.training:__call__] E=7 B=30450	s/sec=2043.91 tok/sec=110055.74 u/sec=24.06	ppl=8.126053 
[2023-12-17:11:43:13:INFO:sockeye.training:__call__] E=7 B=30500	s/sec=1965.38 tok/sec=104777.88 u/sec=24.72	ppl=8.134728 
[2023-12-17:11:43:15:INFO:sockeye.training:__call__] E=7 B=30550	s/sec=2413.08 tok/sec=102308.00 u/sec=25.39	ppl=8.148272 
[2023-12-17:11:43:17:INFO:sockeye.training:__call__] E=7 B=30600	s/sec=2395.01 tok/sec=103903.97 u/sec=25.63	ppl=8.161414 
[2023-12-17:11:43:19:INFO:sockeye.training:__call__] E=7 B=30650	s/sec=2237.37 tok/sec=105242.15 u/sec=25.47	ppl=8.170919 
[2023-12-17:11:43:21:INFO:sockeye.training:__call__] E=7 B=30700	s/sec=2390.42 tok/sec=104087.64 u/sec=25.07	ppl=8.181763 
[2023-12-17:11:43:23:INFO:sockeye.training:__call__] E=7 B=30750	s/sec=2110.92 tok/sec=106381.23 u/sec=24.39	ppl=8.189242 
[2023-12-17:11:43:25:INFO:sockeye.training:__call__] E=7 B=30800	s/sec=2335.60 tok/sec=107670.16 u/sec=24.25	ppl=8.199733 
[2023-12-17:11:43:27:INFO:sockeye.training:__call__] E=7 B=30850	s/sec=2411.78 tok/sec=100627.00 u/sec=25.77	ppl=8.208791 
[2023-12-17:11:43:29:INFO:sockeye.training:__call__] E=7 B=30900	s/sec=2030.79 tok/sec=110406.14 u/sec=24.46	ppl=8.219238 
[2023-12-17:11:43:31:INFO:sockeye.training:__call__] E=7 B=30950	s/sec=2235.90 tok/sec=103210.12 u/sec=24.91	ppl=8.226616 
[2023-12-17:11:43:33:INFO:sockeye.training:__call__] E=7 B=31000	s/sec=2376.98 tok/sec=99617.44 u/sec=24.97	ppl=8.234949 
[2023-12-17:11:43:35:INFO:sockeye.training:__call__] E=7 B=31050	s/sec=2215.12 tok/sec=105520.05 u/sec=25.17	ppl=8.239767 
[2023-12-17:11:43:37:INFO:sockeye.training:__call__] E=7 B=31100	s/sec=2186.14 tok/sec=106270.48 u/sec=24.84	ppl=8.248826 
[2023-12-17:11:43:39:INFO:sockeye.training:__call__] E=7 B=31150	s/sec=2155.67 tok/sec=109904.07 u/sec=24.28	ppl=8.257762 
[2023-12-17:11:43:41:INFO:sockeye.training:__call__] E=7 B=31200	s/sec=2274.00 tok/sec=103602.45 u/sec=25.65	ppl=8.265491 
[2023-12-17:11:43:43:INFO:sockeye.training:__call__] E=7 B=31250	s/sec=2349.64 tok/sec=104309.26 u/sec=25.36	ppl=8.273151 
[2023-12-17:11:43:45:INFO:sockeye.training:__call__] E=7 B=31300	s/sec=2107.22 tok/sec=109038.54 u/sec=24.66	ppl=8.280982 
[2023-12-17:11:43:47:INFO:sockeye.training:__call__] E=7 B=31350	s/sec=2086.93 tok/sec=112708.90 u/sec=22.88	ppl=8.287824 
[2023-12-17:11:43:49:INFO:sockeye.training:__call__] E=7 B=31400	s/sec=2154.79 tok/sec=107331.62 u/sec=24.99	ppl=8.296263 
[2023-12-17:11:43:51:INFO:sockeye.training:__call__] E=7 B=31450	s/sec=2278.97 tok/sec=109899.73 u/sec=24.86	ppl=8.303312 
[2023-12-17:11:43:54:INFO:sockeye.training:__call__] E=7 B=31500	s/sec=2225.87 tok/sec=110153.04 u/sec=24.41	ppl=8.310775 
[2023-12-17:11:43:55:INFO:sockeye.training:__call__] E=7 B=31550	s/sec=2261.96 tok/sec=105960.51 u/sec=25.85	ppl=8.320063 
[2023-12-17:11:43:58:INFO:sockeye.training:__call__] E=7 B=31600	s/sec=2155.71 tok/sec=105380.17 u/sec=23.64	ppl=8.327555 
[2023-12-17:11:44:00:INFO:sockeye.training:__call__] E=7 B=31650	s/sec=2239.21 tok/sec=106404.25 u/sec=24.77	ppl=8.338708 
[2023-12-17:11:44:02:INFO:sockeye.training:__call__] E=7 B=31700	s/sec=2282.55 tok/sec=106064.21 u/sec=25.07	ppl=8.346238 
[2023-12-17:11:44:04:INFO:sockeye.training:__call__] E=7 B=31750	s/sec=2315.13 tok/sec=106616.76 u/sec=25.16	ppl=8.357066 
[2023-12-17:11:44:06:INFO:sockeye.training:__call__] E=7 B=31800	s/sec=2272.13 tok/sec=105619.74 u/sec=25.13	ppl=8.365381 
[2023-12-17:11:44:08:INFO:sockeye.training:__call__] E=7 B=31850	s/sec=2163.21 tok/sec=110178.50 u/sec=24.54	ppl=8.376131 
[2023-12-17:11:44:10:INFO:sockeye.training:__call__] E=7 B=31900	s/sec=2142.63 tok/sec=108463.70 u/sec=25.08	ppl=8.382990 
[2023-12-17:11:44:12:INFO:sockeye.training:__call__] E=7 B=31950	s/sec=2230.00 tok/sec=104533.68 u/sec=25.76	ppl=8.391453 
[2023-12-17:11:44:14:INFO:sockeye.training:__call__] E=8 B=32000	s/sec=2087.27 tok/sec=100337.53 u/sec=23.21	ppl=8.381054 
[2023-12-17:11:44:14:INFO:sockeye.training:_create_checkpoint] Checkpoint [8]	Updates=32000 Epoch=8 Samples=2851920 Time-cost=160.890 Updates/sec=24.862
[2023-12-17:11:44:14:INFO:sockeye.training:_create_checkpoint] Checkpoint [8]	Train-ppl=8.381054
[2023-12-17:11:44:16:INFO:sockeye.training:_evaluate] Checkpoint [8]	Validation-ppl=29.687123	Validation-bleu=0.139074	Validation-chrf=29.068812	Validation-rouge1=0.154156	Validation-rouge2=0.072744	Validation-rougel=0.146566	Validation-length-ratio-mse=1.257084	Validation-ter=96.987206	Validation-avg-sec-per-sent=0.003152	Validation-decode-walltime=1.575938
[2023-12-17:11:44:16:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 3 checkpoints, best so far: 31.983236
[2023-12-17:11:44:16:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [8]	Learning-rate=0.000200
[2023-12-17:11:44:17:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:44:17:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:44:17:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00008"
[2023-12-17:11:44:19:INFO:sockeye.training:__call__] E=8 B=32050	s/sec=904.51 tok/sec=38369.44 u/sec=9.50	ppl=7.374685 
[2023-12-17:11:44:21:INFO:sockeye.training:__call__] E=8 B=32100	s/sec=2267.83 tok/sec=107134.28 u/sec=24.91	ppl=7.284335 
[2023-12-17:11:44:23:INFO:sockeye.training:__call__] E=8 B=32150	s/sec=2295.09 tok/sec=104219.32 u/sec=25.66	ppl=7.288698 
[2023-12-17:11:44:25:INFO:sockeye.training:__call__] E=8 B=32200	s/sec=2056.30 tok/sec=109596.87 u/sec=24.25	ppl=7.212497 
[2023-12-17:11:44:27:INFO:sockeye.training:__call__] E=8 B=32250	s/sec=2095.02 tok/sec=108320.70 u/sec=24.85	ppl=7.241880 
[2023-12-17:11:44:29:INFO:sockeye.training:__call__] E=8 B=32300	s/sec=2178.49 tok/sec=110585.90 u/sec=24.49	ppl=7.262406 
[2023-12-17:11:44:31:INFO:sockeye.training:__call__] E=8 B=32350	s/sec=2231.80 tok/sec=105494.31 u/sec=25.50	ppl=7.276950 
[2023-12-17:11:44:33:INFO:sockeye.training:__call__] E=8 B=32400	s/sec=2296.81 tok/sec=106663.31 u/sec=25.45	ppl=7.278899 
[2023-12-17:11:44:35:INFO:sockeye.training:__call__] E=8 B=32450	s/sec=2167.74 tok/sec=108274.34 u/sec=25.18	ppl=7.285471 
[2023-12-17:11:44:37:INFO:sockeye.training:__call__] E=8 B=32500	s/sec=2442.48 tok/sec=104201.35 u/sec=25.66	ppl=7.309207 
[2023-12-17:11:44:39:INFO:sockeye.training:__call__] E=8 B=32550	s/sec=2337.99 tok/sec=103460.21 u/sec=25.55	ppl=7.329518 
[2023-12-17:11:44:41:INFO:sockeye.training:__call__] E=8 B=32600	s/sec=1990.31 tok/sec=109944.08 u/sec=24.63	ppl=7.342921 
[2023-12-17:11:44:43:INFO:sockeye.training:__call__] E=8 B=32650	s/sec=2401.68 tok/sec=108222.06 u/sec=24.73	ppl=7.367640 
[2023-12-17:11:44:45:INFO:sockeye.training:__call__] E=8 B=32700	s/sec=2366.26 tok/sec=105747.18 u/sec=25.50	ppl=7.377775 
[2023-12-17:11:44:47:INFO:sockeye.training:__call__] E=8 B=32750	s/sec=2181.57 tok/sec=105101.57 u/sec=25.16	ppl=7.391007 
[2023-12-17:11:44:49:INFO:sockeye.training:__call__] E=8 B=32800	s/sec=2129.06 tok/sec=106388.43 u/sec=25.20	ppl=7.398178 
[2023-12-17:11:44:51:INFO:sockeye.training:__call__] E=8 B=32850	s/sec=2088.63 tok/sec=110558.15 u/sec=24.17	ppl=7.414154 
[2023-12-17:11:44:53:INFO:sockeye.training:__call__] E=8 B=32900	s/sec=2449.00 tok/sec=105126.15 u/sec=25.68	ppl=7.433600 
[2023-12-17:11:44:55:INFO:sockeye.training:__call__] E=8 B=32950	s/sec=1958.42 tok/sec=119892.10 u/sec=22.88	ppl=7.445136 
[2023-12-17:11:44:57:INFO:sockeye.training:__call__] E=8 B=33000	s/sec=2189.52 tok/sec=110310.37 u/sec=24.84	ppl=7.452323 
[2023-12-17:11:44:59:INFO:sockeye.training:__call__] E=8 B=33050	s/sec=2209.09 tok/sec=106817.64 u/sec=25.43	ppl=7.468690 
[2023-12-17:11:45:01:INFO:sockeye.training:__call__] E=8 B=33100	s/sec=2160.02 tok/sec=111180.61 u/sec=24.41	ppl=7.477354 
[2023-12-17:11:45:03:INFO:sockeye.training:__call__] E=8 B=33150	s/sec=2047.99 tok/sec=110771.34 u/sec=24.47	ppl=7.485241 
[2023-12-17:11:45:05:INFO:sockeye.training:__call__] E=8 B=33200	s/sec=2116.29 tok/sec=109556.66 u/sec=24.72	ppl=7.500502 
[2023-12-17:11:45:07:INFO:sockeye.training:__call__] E=8 B=33250	s/sec=2124.42 tok/sec=109008.81 u/sec=24.54	ppl=7.512975 
[2023-12-17:11:45:09:INFO:sockeye.training:__call__] E=8 B=33300	s/sec=2217.95 tok/sec=108068.88 u/sec=25.11	ppl=7.526637 
[2023-12-17:11:45:11:INFO:sockeye.training:__call__] E=8 B=33350	s/sec=2057.30 tok/sec=110809.42 u/sec=24.73	ppl=7.537450 
[2023-12-17:11:45:13:INFO:sockeye.training:__call__] E=8 B=33400	s/sec=2472.22 tok/sec=101215.46 u/sec=25.97	ppl=7.550631 
[2023-12-17:11:45:15:INFO:sockeye.training:__call__] E=8 B=33450	s/sec=2194.30 tok/sec=105654.42 u/sec=25.63	ppl=7.558947 
[2023-12-17:11:45:17:INFO:sockeye.training:__call__] E=8 B=33500	s/sec=2025.82 tok/sec=110047.41 u/sec=24.59	ppl=7.567783 
[2023-12-17:11:45:19:INFO:sockeye.training:__call__] E=8 B=33550	s/sec=2144.01 tok/sec=108352.45 u/sec=25.00	ppl=7.573907 
[2023-12-17:11:45:21:INFO:sockeye.training:__call__] E=8 B=33600	s/sec=2141.93 tok/sec=109799.75 u/sec=24.70	ppl=7.577943 
[2023-12-17:11:45:23:INFO:sockeye.training:__call__] E=8 B=33650	s/sec=2480.21 tok/sec=101898.99 u/sec=26.27	ppl=7.593129 
[2023-12-17:11:45:25:INFO:sockeye.training:__call__] E=8 B=33700	s/sec=2355.58 tok/sec=103904.53 u/sec=25.30	ppl=7.610253 
[2023-12-17:11:45:27:INFO:sockeye.training:__call__] E=8 B=33750	s/sec=2411.25 tok/sec=104787.41 u/sec=25.54	ppl=7.623166 
[2023-12-17:11:45:29:INFO:sockeye.training:__call__] E=8 B=33800	s/sec=2325.87 tok/sec=105769.10 u/sec=24.93	ppl=7.636165 
[2023-12-17:11:45:31:INFO:sockeye.training:__call__] E=8 B=33850	s/sec=2141.21 tok/sec=107371.48 u/sec=24.92	ppl=7.642625 
[2023-12-17:11:45:33:INFO:sockeye.training:__call__] E=8 B=33900	s/sec=2343.73 tok/sec=104398.79 u/sec=25.21	ppl=7.653104 
[2023-12-17:11:45:35:INFO:sockeye.training:__call__] E=8 B=33950	s/sec=2221.58 tok/sec=108114.38 u/sec=24.58	ppl=7.666198 
[2023-12-17:11:45:37:INFO:sockeye.training:__call__] E=8 B=34000	s/sec=2121.72 tok/sec=110876.61 u/sec=24.42	ppl=7.675919 
[2023-12-17:11:45:39:INFO:sockeye.training:__call__] E=8 B=34050	s/sec=1983.32 tok/sec=109896.67 u/sec=24.69	ppl=7.685378 
[2023-12-17:11:45:41:INFO:sockeye.training:__call__] E=8 B=34100	s/sec=2325.83 tok/sec=106186.16 u/sec=25.06	ppl=7.694672 
[2023-12-17:11:45:43:INFO:sockeye.training:__call__] E=8 B=34150	s/sec=2513.55 tok/sec=104013.65 u/sec=25.59	ppl=7.710421 
[2023-12-17:11:45:45:INFO:sockeye.training:__call__] E=8 B=34200	s/sec=2174.26 tok/sec=106538.89 u/sec=24.98	ppl=7.719820 
[2023-12-17:11:45:47:INFO:sockeye.training:__call__] E=8 B=34250	s/sec=2172.42 tok/sec=108754.97 u/sec=24.64	ppl=7.728479 
[2023-12-17:11:45:49:INFO:sockeye.training:__call__] E=8 B=34300	s/sec=2302.98 tok/sec=105584.08 u/sec=25.66	ppl=7.741052 
[2023-12-17:11:45:51:INFO:sockeye.training:__call__] E=8 B=34350	s/sec=2273.91 tok/sec=106404.34 u/sec=25.52	ppl=7.753390 
[2023-12-17:11:45:53:INFO:sockeye.training:__call__] E=8 B=34400	s/sec=2150.00 tok/sec=109397.30 u/sec=24.70	ppl=7.764252 
[2023-12-17:11:45:55:INFO:sockeye.training:__call__] E=8 B=34450	s/sec=2318.11 tok/sec=105618.26 u/sec=25.37	ppl=7.774273 
[2023-12-17:11:45:57:INFO:sockeye.training:__call__] E=8 B=34500	s/sec=2323.87 tok/sec=106451.95 u/sec=25.35	ppl=7.786409 
[2023-12-17:11:45:59:INFO:sockeye.training:__call__] E=8 B=34550	s/sec=1986.10 tok/sec=109641.89 u/sec=24.20	ppl=7.796474 
[2023-12-17:11:46:01:INFO:sockeye.training:__call__] E=8 B=34600	s/sec=2404.93 tok/sec=105319.11 u/sec=25.09	ppl=7.807753 
[2023-12-17:11:46:03:INFO:sockeye.training:__call__] E=8 B=34650	s/sec=2483.78 tok/sec=105513.31 u/sec=25.74	ppl=7.818553 
[2023-12-17:11:46:05:INFO:sockeye.training:__call__] E=8 B=34700	s/sec=2306.19 tok/sec=104849.07 u/sec=25.24	ppl=7.828945 
[2023-12-17:11:46:07:INFO:sockeye.training:__call__] E=8 B=34750	s/sec=2227.28 tok/sec=104957.39 u/sec=24.59	ppl=7.835011 
[2023-12-17:11:46:09:INFO:sockeye.training:__call__] E=8 B=34800	s/sec=2247.54 tok/sec=108937.85 u/sec=24.99	ppl=7.844035 
[2023-12-17:11:46:11:INFO:sockeye.training:__call__] E=8 B=34850	s/sec=2300.64 tok/sec=104107.94 u/sec=25.86	ppl=7.856753 
[2023-12-17:11:46:13:INFO:sockeye.training:__call__] E=8 B=34900	s/sec=2259.84 tok/sec=104210.78 u/sec=26.01	ppl=7.866614 
[2023-12-17:11:46:15:INFO:sockeye.training:__call__] E=8 B=34950	s/sec=2246.56 tok/sec=109018.07 u/sec=24.98	ppl=7.876000 
[2023-12-17:11:46:17:INFO:sockeye.training:__call__] E=8 B=35000	s/sec=2252.44 tok/sec=106056.97 u/sec=25.55	ppl=7.883852 
[2023-12-17:11:46:19:INFO:sockeye.training:__call__] E=8 B=35050	s/sec=2236.60 tok/sec=109394.21 u/sec=24.87	ppl=7.891437 
[2023-12-17:11:46:21:INFO:sockeye.training:__call__] E=8 B=35100	s/sec=2149.32 tok/sec=109725.37 u/sec=24.51	ppl=7.896862 
[2023-12-17:11:46:23:INFO:sockeye.training:__call__] E=8 B=35150	s/sec=2363.24 tok/sec=107357.47 u/sec=24.99	ppl=7.908335 
[2023-12-17:11:46:25:INFO:sockeye.training:__call__] E=8 B=35200	s/sec=2072.21 tok/sec=109437.79 u/sec=24.58	ppl=7.915861 
[2023-12-17:11:46:27:INFO:sockeye.training:__call__] E=8 B=35250	s/sec=2115.16 tok/sec=109511.43 u/sec=24.85	ppl=7.921853 
[2023-12-17:11:46:29:INFO:sockeye.training:__call__] E=8 B=35300	s/sec=2449.75 tok/sec=103281.35 u/sec=25.52	ppl=7.932048 
[2023-12-17:11:46:31:INFO:sockeye.training:__call__] E=8 B=35350	s/sec=1971.63 tok/sec=110537.37 u/sec=24.65	ppl=7.937140 
[2023-12-17:11:46:33:INFO:sockeye.training:__call__] E=8 B=35400	s/sec=2333.34 tok/sec=105895.75 u/sec=25.10	ppl=7.944150 
[2023-12-17:11:46:35:INFO:sockeye.training:__call__] E=8 B=35450	s/sec=2463.28 tok/sec=101565.62 u/sec=26.05	ppl=7.956362 
[2023-12-17:11:46:37:INFO:sockeye.training:__call__] E=8 B=35500	s/sec=2398.71 tok/sec=101852.95 u/sec=25.63	ppl=7.967422 
[2023-12-17:11:46:39:INFO:sockeye.training:__call__] E=8 B=35550	s/sec=1936.18 tok/sec=107694.40 u/sec=24.85	ppl=7.973260 
[2023-12-17:11:46:41:INFO:sockeye.training:__call__] E=8 B=35600	s/sec=2098.54 tok/sec=109266.52 u/sec=24.70	ppl=7.979335 
[2023-12-17:11:46:43:INFO:sockeye.training:__call__] E=8 B=35650	s/sec=2445.85 tok/sec=104683.84 u/sec=25.61	ppl=7.988933 
[2023-12-17:11:46:45:INFO:sockeye.training:__call__] E=8 B=35700	s/sec=2358.00 tok/sec=105471.18 u/sec=25.11	ppl=7.996464 
[2023-12-17:11:46:47:INFO:sockeye.training:__call__] E=8 B=35750	s/sec=2317.17 tok/sec=104654.77 u/sec=25.41	ppl=8.004386 
[2023-12-17:11:46:49:INFO:sockeye.training:__call__] E=8 B=35800	s/sec=2026.88 tok/sec=106312.72 u/sec=25.09	ppl=8.011009 
[2023-12-17:11:46:51:INFO:sockeye.training:__call__] E=8 B=35850	s/sec=2020.28 tok/sec=109798.22 u/sec=24.14	ppl=8.017231 
[2023-12-17:11:46:53:INFO:sockeye.training:__call__] E=8 B=35900	s/sec=2369.34 tok/sec=104423.92 u/sec=24.56	ppl=8.026023 
[2023-12-17:11:46:55:INFO:sockeye.training:__call__] E=8 B=35950	s/sec=2265.89 tok/sec=107645.57 u/sec=24.93	ppl=8.036550 
[2023-12-17:11:46:57:INFO:sockeye.training:__call__] E=9 B=36000	s/sec=1890.98 tok/sec=105238.49 u/sec=23.31	ppl=8.023116 
[2023-12-17:11:46:57:INFO:sockeye.training:_create_checkpoint] Checkpoint [9]	Updates=36000 Epoch=9 Samples=3208040 Time-cost=159.915 Updates/sec=25.013
[2023-12-17:11:46:57:INFO:sockeye.training:_create_checkpoint] Checkpoint [9]	Train-ppl=8.023116
[2023-12-17:11:46:59:INFO:sockeye.training:_evaluate] Checkpoint [9]	Validation-ppl=29.884377	Validation-bleu=0.165922	Validation-chrf=31.158840	Validation-rouge1=0.163998	Validation-rouge2=0.081888	Validation-rougel=0.156817	Validation-length-ratio-mse=1.333691	Validation-ter=97.606273	Validation-avg-sec-per-sent=0.003181	Validation-decode-walltime=1.590617
[2023-12-17:11:46:59:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 4 checkpoints, best so far: 31.983236
[2023-12-17:11:46:59:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [9]	Learning-rate=0.000200
[2023-12-17:11:47:00:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:47:00:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:47:00:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00009"
[2023-12-17:11:47:02:INFO:sockeye.training:__call__] E=9 B=36050	s/sec=804.87 tok/sec=40009.21 u/sec=9.26	ppl=6.901913 
[2023-12-17:11:47:04:INFO:sockeye.training:__call__] E=9 B=36100	s/sec=1918.08 tok/sec=111871.11 u/sec=24.02	ppl=6.890963 
[2023-12-17:11:47:06:INFO:sockeye.training:__call__] E=9 B=36150	s/sec=2521.67 tok/sec=103196.99 u/sec=26.01	ppl=6.962972 
[2023-12-17:11:47:08:INFO:sockeye.training:__call__] E=9 B=36200	s/sec=2211.00 tok/sec=107713.08 u/sec=24.76	ppl=6.987483 
[2023-12-17:11:47:10:INFO:sockeye.training:__call__] E=9 B=36250	s/sec=2061.33 tok/sec=109672.06 u/sec=24.87	ppl=6.986610 
[2023-12-17:11:47:12:INFO:sockeye.training:__call__] E=9 B=36300	s/sec=2334.34 tok/sec=106721.82 u/sec=24.77	ppl=6.997938 
[2023-12-17:11:47:14:INFO:sockeye.training:__call__] E=9 B=36350	s/sec=2279.76 tok/sec=109176.20 u/sec=24.65	ppl=7.018161 
[2023-12-17:11:47:16:INFO:sockeye.training:__call__] E=9 B=36400	s/sec=2295.83 tok/sec=105931.33 u/sec=25.26	ppl=7.040086 
[2023-12-17:11:47:18:INFO:sockeye.training:__call__] E=9 B=36450	s/sec=2024.88 tok/sec=110582.67 u/sec=24.67	ppl=7.045817 
[2023-12-17:11:47:20:INFO:sockeye.training:__call__] E=9 B=36500	s/sec=2250.63 tok/sec=104854.86 u/sec=24.81	ppl=7.068834 
[2023-12-17:11:47:22:INFO:sockeye.training:__call__] E=9 B=36550	s/sec=2231.85 tok/sec=108862.69 u/sec=25.09	ppl=7.084538 
[2023-12-17:11:47:24:INFO:sockeye.training:__call__] E=9 B=36600	s/sec=2358.45 tok/sec=102938.12 u/sec=25.55	ppl=7.107569 
[2023-12-17:11:47:26:INFO:sockeye.training:__call__] E=9 B=36650	s/sec=2292.74 tok/sec=105319.04 u/sec=25.50	ppl=7.120186 
[2023-12-17:11:47:28:INFO:sockeye.training:__call__] E=9 B=36700	s/sec=2274.71 tok/sec=106003.28 u/sec=25.12	ppl=7.139070 
[2023-12-17:11:47:30:INFO:sockeye.training:__call__] E=9 B=36750	s/sec=2050.33 tok/sec=109520.30 u/sec=24.79	ppl=7.154487 
[2023-12-17:11:47:32:INFO:sockeye.training:__call__] E=9 B=36800	s/sec=2099.15 tok/sec=110676.44 u/sec=24.75	ppl=7.163981 
[2023-12-17:11:47:34:INFO:sockeye.training:__call__] E=9 B=36850	s/sec=2258.85 tok/sec=107584.26 u/sec=25.26	ppl=7.171898 
[2023-12-17:11:47:36:INFO:sockeye.training:__call__] E=9 B=36900	s/sec=2175.86 tok/sec=105101.32 u/sec=24.55	ppl=7.193922 
[2023-12-17:11:47:38:INFO:sockeye.training:__call__] E=9 B=36950	s/sec=2301.79 tok/sec=105034.36 u/sec=24.76	ppl=7.203685 
[2023-12-17:11:47:40:INFO:sockeye.training:__call__] E=9 B=37000	s/sec=1949.20 tok/sec=108176.69 u/sec=24.66	ppl=7.204760 
[2023-12-17:11:47:42:INFO:sockeye.training:__call__] E=9 B=37050	s/sec=2080.06 tok/sec=110415.01 u/sec=24.44	ppl=7.213642 
[2023-12-17:11:47:44:INFO:sockeye.training:__call__] E=9 B=37100	s/sec=2277.82 tok/sec=107314.16 u/sec=25.06	ppl=7.221283 
[2023-12-17:11:47:46:INFO:sockeye.training:__call__] E=9 B=37150	s/sec=2176.71 tok/sec=109835.47 u/sec=24.60	ppl=7.230839 
[2023-12-17:11:47:48:INFO:sockeye.training:__call__] E=9 B=37200	s/sec=2229.33 tok/sec=108800.93 u/sec=25.01	ppl=7.239676 
[2023-12-17:11:47:51:INFO:sockeye.training:__call__] E=9 B=37250	s/sec=1930.66 tok/sec=112233.57 u/sec=24.18	ppl=7.240490 
[2023-12-17:11:47:53:INFO:sockeye.training:__call__] E=9 B=37300	s/sec=2182.48 tok/sec=105471.51 u/sec=25.31	ppl=7.249252 
[2023-12-17:11:47:55:INFO:sockeye.training:__call__] E=9 B=37350	s/sec=2165.72 tok/sec=108776.38 u/sec=25.11	ppl=7.262040 
[2023-12-17:11:47:57:INFO:sockeye.training:__call__] E=9 B=37400	s/sec=2287.23 tok/sec=106699.70 u/sec=25.12	ppl=7.274979 
[2023-12-17:11:47:58:INFO:sockeye.training:__call__] E=9 B=37450	s/sec=2381.93 tok/sec=102971.46 u/sec=25.58	ppl=7.286144 
[2023-12-17:11:48:00:INFO:sockeye.training:__call__] E=9 B=37500	s/sec=2291.64 tok/sec=103810.81 u/sec=25.71	ppl=7.299672 
[2023-12-17:11:48:02:INFO:sockeye.training:__call__] E=9 B=37550	s/sec=2341.87 tok/sec=104138.47 u/sec=25.95	ppl=7.312457 
[2023-12-17:11:48:04:INFO:sockeye.training:__call__] E=9 B=37600	s/sec=2119.28 tok/sec=114961.77 u/sec=23.24	ppl=7.321501 
[2023-12-17:11:48:06:INFO:sockeye.training:__call__] E=9 B=37650	s/sec=2268.67 tok/sec=105572.61 u/sec=25.46	ppl=7.333908 
[2023-12-17:11:48:08:INFO:sockeye.training:__call__] E=9 B=37700	s/sec=2430.13 tok/sec=102977.75 u/sec=25.92	ppl=7.347161 
[2023-12-17:11:48:10:INFO:sockeye.training:__call__] E=9 B=37750	s/sec=2179.97 tok/sec=109435.42 u/sec=24.82	ppl=7.359585 
[2023-12-17:11:48:12:INFO:sockeye.training:__call__] E=9 B=37800	s/sec=2404.58 tok/sec=104002.11 u/sec=25.56	ppl=7.370550 
[2023-12-17:11:48:14:INFO:sockeye.training:__call__] E=9 B=37850	s/sec=1930.10 tok/sec=110004.09 u/sec=24.57	ppl=7.374790 
[2023-12-17:11:48:16:INFO:sockeye.training:__call__] E=9 B=37900	s/sec=2296.28 tok/sec=104899.50 u/sec=25.05	ppl=7.385159 
[2023-12-17:11:48:18:INFO:sockeye.training:__call__] E=9 B=37950	s/sec=2380.82 tok/sec=105345.32 u/sec=25.57	ppl=7.396392 
[2023-12-17:11:48:20:INFO:sockeye.training:__call__] E=9 B=38000	s/sec=2183.59 tok/sec=107326.22 u/sec=24.86	ppl=7.408460 
[2023-12-17:11:48:22:INFO:sockeye.training:__call__] E=9 B=38050	s/sec=2076.09 tok/sec=107612.16 u/sec=24.48	ppl=7.417181 
[2023-12-17:11:48:24:INFO:sockeye.training:__call__] E=9 B=38100	s/sec=2421.80 tok/sec=102769.94 u/sec=25.74	ppl=7.426840 
[2023-12-17:11:48:26:INFO:sockeye.training:__call__] E=9 B=38150	s/sec=2594.11 tok/sec=101949.91 u/sec=25.78	ppl=7.436822 
[2023-12-17:11:48:28:INFO:sockeye.training:__call__] E=9 B=38200	s/sec=2216.76 tok/sec=109439.87 u/sec=24.96	ppl=7.450195 
[2023-12-17:11:48:30:INFO:sockeye.training:__call__] E=9 B=38250	s/sec=2021.46 tok/sec=112272.12 u/sec=24.20	ppl=7.455019 
[2023-12-17:11:48:32:INFO:sockeye.training:__call__] E=9 B=38300	s/sec=2325.24 tok/sec=102817.09 u/sec=25.54	ppl=7.465732 
[2023-12-17:11:48:34:INFO:sockeye.training:__call__] E=9 B=38350	s/sec=2479.12 tok/sec=102399.34 u/sec=25.53	ppl=7.477238 
[2023-12-17:11:48:36:INFO:sockeye.training:__call__] E=9 B=38400	s/sec=2183.74 tok/sec=106348.50 u/sec=25.42	ppl=7.483660 
[2023-12-17:11:48:38:INFO:sockeye.training:__call__] E=9 B=38450	s/sec=2188.70 tok/sec=108941.31 u/sec=24.87	ppl=7.491617 
[2023-12-17:11:48:40:INFO:sockeye.training:__call__] E=9 B=38500	s/sec=2356.10 tok/sec=105437.58 u/sec=25.13	ppl=7.501322 
[2023-12-17:11:48:42:INFO:sockeye.training:__call__] E=9 B=38550	s/sec=2459.93 tok/sec=106134.22 u/sec=25.08	ppl=7.511101 
[2023-12-17:11:48:44:INFO:sockeye.training:__call__] E=9 B=38600	s/sec=1945.89 tok/sec=111206.86 u/sec=24.28	ppl=7.518937 
[2023-12-17:11:48:46:INFO:sockeye.training:__call__] E=9 B=38650	s/sec=2270.30 tok/sec=106759.41 u/sec=24.72	ppl=7.530249 
[2023-12-17:11:48:48:INFO:sockeye.training:__call__] E=9 B=38700	s/sec=2024.03 tok/sec=108620.30 u/sec=24.61	ppl=7.537113 
[2023-12-17:11:48:50:INFO:sockeye.training:__call__] E=9 B=38750	s/sec=2343.72 tok/sec=104005.55 u/sec=25.52	ppl=7.546878 
[2023-12-17:11:48:52:INFO:sockeye.training:__call__] E=9 B=38800	s/sec=2523.91 tok/sec=103451.79 u/sec=25.65	ppl=7.556851 
[2023-12-17:11:48:54:INFO:sockeye.training:__call__] E=9 B=38850	s/sec=2203.42 tok/sec=108607.26 u/sec=24.68	ppl=7.564461 
[2023-12-17:11:48:56:INFO:sockeye.training:__call__] E=9 B=38900	s/sec=2237.76 tok/sec=106353.87 u/sec=25.06	ppl=7.572251 
[2023-12-17:11:48:58:INFO:sockeye.training:__call__] E=9 B=38950	s/sec=2187.94 tok/sec=107045.91 u/sec=25.51	ppl=7.581315 
[2023-12-17:11:49:00:INFO:sockeye.training:__call__] E=9 B=39000	s/sec=2284.80 tok/sec=106532.38 u/sec=25.27	ppl=7.591142 
[2023-12-17:11:49:02:INFO:sockeye.training:__call__] E=9 B=39050	s/sec=2252.92 tok/sec=103937.90 u/sec=25.65	ppl=7.599432 
[2023-12-17:11:49:04:INFO:sockeye.training:__call__] E=9 B=39100	s/sec=2527.24 tok/sec=102657.51 u/sec=26.06	ppl=7.610139 
[2023-12-17:11:49:06:INFO:sockeye.training:__call__] E=9 B=39150	s/sec=2325.67 tok/sec=103973.11 u/sec=25.68	ppl=7.618956 
[2023-12-17:11:49:08:INFO:sockeye.training:__call__] E=9 B=39200	s/sec=2369.86 tok/sec=107046.90 u/sec=25.19	ppl=7.629259 
[2023-12-17:11:49:10:INFO:sockeye.training:__call__] E=9 B=39250	s/sec=2429.04 tok/sec=104496.90 u/sec=25.13	ppl=7.638338 
[2023-12-17:11:49:12:INFO:sockeye.training:__call__] E=9 B=39300	s/sec=1991.37 tok/sec=109195.83 u/sec=24.50	ppl=7.646165 
[2023-12-17:11:49:14:INFO:sockeye.training:__call__] E=9 B=39350	s/sec=1785.10 tok/sec=113973.02 u/sec=23.74	ppl=7.649940 
[2023-12-17:11:49:16:INFO:sockeye.training:__call__] E=9 B=39400	s/sec=2226.87 tok/sec=108456.88 u/sec=24.99	ppl=7.657611 
[2023-12-17:11:49:18:INFO:sockeye.training:__call__] E=9 B=39450	s/sec=2240.43 tok/sec=109903.20 u/sec=24.61	ppl=7.664642 
[2023-12-17:11:49:20:INFO:sockeye.training:__call__] E=9 B=39500	s/sec=2063.90 tok/sec=107258.41 u/sec=25.19	ppl=7.671282 
[2023-12-17:11:49:22:INFO:sockeye.training:__call__] E=9 B=39550	s/sec=2350.61 tok/sec=107040.84 u/sec=24.99	ppl=7.680298 
[2023-12-17:11:49:24:INFO:sockeye.training:__call__] E=9 B=39600	s/sec=2402.37 tok/sec=106172.49 u/sec=25.23	ppl=7.690274 
[2023-12-17:11:49:26:INFO:sockeye.training:__call__] E=9 B=39650	s/sec=2367.51 tok/sec=105469.09 u/sec=24.74	ppl=7.697810 
[2023-12-17:11:49:28:INFO:sockeye.training:__call__] E=9 B=39700	s/sec=2382.51 tok/sec=104879.14 u/sec=25.28	ppl=7.706471 
[2023-12-17:11:49:30:INFO:sockeye.training:__call__] E=9 B=39750	s/sec=2196.58 tok/sec=106592.26 u/sec=25.10	ppl=7.714518 
[2023-12-17:11:49:32:INFO:sockeye.training:__call__] E=9 B=39800	s/sec=2176.05 tok/sec=104803.78 u/sec=25.09	ppl=7.720561 
[2023-12-17:11:49:34:INFO:sockeye.training:__call__] E=9 B=39850	s/sec=2002.38 tok/sec=109514.86 u/sec=24.07	ppl=7.726653 
[2023-12-17:11:49:36:INFO:sockeye.training:__call__] E=9 B=39900	s/sec=2230.10 tok/sec=106205.98 u/sec=25.20	ppl=7.734975 
[2023-12-17:11:49:38:INFO:sockeye.training:__call__] E=9 B=39950	s/sec=2268.57 tok/sec=106027.69 u/sec=25.14	ppl=7.743428 
[2023-12-17:11:49:40:INFO:sockeye.training:__call__] E=10 B=40000	s/sec=2167.39 tok/sec=102367.64 u/sec=23.68	ppl=7.731083 
[2023-12-17:11:49:40:INFO:sockeye.training:_create_checkpoint] Checkpoint [10]	Updates=40000 Epoch=10 Samples=3564976 Time-cost=160.041 Updates/sec=24.994
[2023-12-17:11:49:40:INFO:sockeye.training:_create_checkpoint] Checkpoint [10]	Train-ppl=7.731083
[2023-12-17:11:49:43:INFO:sockeye.training:_evaluate] Checkpoint [10]	Validation-ppl=31.639580	Validation-bleu=0.172840	Validation-chrf=31.113908	Validation-rouge1=0.156950	Validation-rouge2=0.075279	Validation-rougel=0.149662	Validation-length-ratio-mse=1.303005	Validation-ter=96.657037	Validation-avg-sec-per-sent=0.003251	Validation-decode-walltime=1.625525
[2023-12-17:11:49:43:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 5 checkpoints, best so far: 31.983236
[2023-12-17:11:49:43:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [10]	Learning-rate=0.000200
[2023-12-17:11:49:43:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:49:43:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=5, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:49:44:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00010"
[2023-12-17:11:49:46:INFO:sockeye.training:__call__] E=10 B=40050	s/sec=837.17 tok/sec=41102.51 u/sec=9.57	ppl=6.775582 
[2023-12-17:11:49:48:INFO:sockeye.training:__call__] E=10 B=40100	s/sec=2259.61 tok/sec=105145.73 u/sec=24.91	ppl=6.785598 
[2023-12-17:11:49:50:INFO:sockeye.training:__call__] E=10 B=40150	s/sec=2512.10 tok/sec=103008.44 u/sec=25.28	ppl=6.823666 
[2023-12-17:11:49:52:INFO:sockeye.training:__call__] E=10 B=40200	s/sec=2147.83 tok/sec=109108.35 u/sec=24.72	ppl=6.822364 
[2023-12-17:11:49:54:INFO:sockeye.training:__call__] E=10 B=40250	s/sec=1909.03 tok/sec=110229.76 u/sec=24.50	ppl=6.801383 
[2023-12-17:11:49:56:INFO:sockeye.training:__call__] E=10 B=40300	s/sec=2495.55 tok/sec=101491.24 u/sec=25.57	ppl=6.826864 
[2023-12-17:11:49:58:INFO:sockeye.training:__call__] E=10 B=40350	s/sec=2029.13 tok/sec=110535.03 u/sec=24.44	ppl=6.819743 
[2023-12-17:11:50:00:INFO:sockeye.training:__call__] E=10 B=40400	s/sec=2295.18 tok/sec=104650.31 u/sec=25.21	ppl=6.832038 
[2023-12-17:11:50:02:INFO:sockeye.training:__call__] E=10 B=40450	s/sec=2020.73 tok/sec=110445.48 u/sec=24.67	ppl=6.834407 
[2023-12-17:11:50:04:INFO:sockeye.training:__call__] E=10 B=40500	s/sec=2132.76 tok/sec=109115.79 u/sec=24.50	ppl=6.837348 
[2023-12-17:11:50:06:INFO:sockeye.training:__call__] E=10 B=40550	s/sec=2284.36 tok/sec=102692.01 u/sec=26.01	ppl=6.851905 
[2023-12-17:11:50:08:INFO:sockeye.training:__call__] E=10 B=40600	s/sec=2266.64 tok/sec=108120.35 u/sec=24.09	ppl=6.858550 
[2023-12-17:11:50:10:INFO:sockeye.training:__call__] E=10 B=40650	s/sec=2016.86 tok/sec=106503.09 u/sec=25.16	ppl=6.869249 
[2023-12-17:11:50:12:INFO:sockeye.training:__call__] E=10 B=40700	s/sec=2193.91 tok/sec=106146.51 u/sec=25.21	ppl=6.885398 
[2023-12-17:11:50:14:INFO:sockeye.training:__call__] E=10 B=40750	s/sec=1827.11 tok/sec=109260.53 u/sec=24.88	ppl=6.884065 
[2023-12-17:11:50:16:INFO:sockeye.training:__call__] E=10 B=40800	s/sec=2209.38 tok/sec=105798.39 u/sec=24.61	ppl=6.901469 
[2023-12-17:11:50:18:INFO:sockeye.training:__call__] E=10 B=40850	s/sec=2615.25 tok/sec=103104.01 u/sec=25.58	ppl=6.919566 
[2023-12-17:11:50:20:INFO:sockeye.training:__call__] E=10 B=40900	s/sec=2322.61 tok/sec=102293.76 u/sec=25.74	ppl=6.931080 
[2023-12-17:11:50:22:INFO:sockeye.training:__call__] E=10 B=40950	s/sec=2370.52 tok/sec=102276.62 u/sec=25.33	ppl=6.947948 
[2023-12-17:11:50:24:INFO:sockeye.training:__call__] E=10 B=41000	s/sec=2161.95 tok/sec=109209.36 u/sec=24.13	ppl=6.960431 
[2023-12-17:11:50:26:INFO:sockeye.training:__call__] E=10 B=41050	s/sec=2031.92 tok/sec=108379.77 u/sec=24.56	ppl=6.964306 
[2023-12-17:11:50:28:INFO:sockeye.training:__call__] E=10 B=41100	s/sec=2362.53 tok/sec=103951.39 u/sec=25.46	ppl=6.983764 
[2023-12-17:11:50:30:INFO:sockeye.training:__call__] E=10 B=41150	s/sec=2187.38 tok/sec=109330.22 u/sec=24.37	ppl=6.996544 
[2023-12-17:11:50:32:INFO:sockeye.training:__call__] E=10 B=41200	s/sec=2695.85 tok/sec=99274.05 u/sec=26.12	ppl=7.018462 
[2023-12-17:11:50:34:INFO:sockeye.training:__call__] E=10 B=41250	s/sec=2291.45 tok/sec=106619.36 u/sec=24.61	ppl=7.030278 
[2023-12-17:11:50:36:INFO:sockeye.training:__call__] E=10 B=41300	s/sec=2286.89 tok/sec=103225.38 u/sec=24.94	ppl=7.043053 
[2023-12-17:11:50:38:INFO:sockeye.training:__call__] E=10 B=41350	s/sec=2377.58 tok/sec=104187.02 u/sec=25.62	ppl=7.053045 
[2023-12-17:11:50:40:INFO:sockeye.training:__call__] E=10 B=41400	s/sec=2522.30 tok/sec=102463.79 u/sec=25.06	ppl=7.069603 
[2023-12-17:11:50:42:INFO:sockeye.training:__call__] E=10 B=41450	s/sec=2250.81 tok/sec=105100.97 u/sec=24.68	ppl=7.081377 
[2023-12-17:11:50:44:INFO:sockeye.training:__call__] E=10 B=41500	s/sec=2096.92 tok/sec=108849.30 u/sec=24.82	ppl=7.086101 
[2023-12-17:11:50:46:INFO:sockeye.training:__call__] E=10 B=41550	s/sec=2347.85 tok/sec=105689.28 u/sec=25.00	ppl=7.097213 
[2023-12-17:11:50:48:INFO:sockeye.training:__call__] E=10 B=41600	s/sec=1990.49 tok/sec=108998.03 u/sec=24.73	ppl=7.103843 
[2023-12-17:11:50:50:INFO:sockeye.training:__call__] E=10 B=41650	s/sec=2029.12 tok/sec=107257.81 u/sec=24.77	ppl=7.112308 
[2023-12-17:11:50:52:INFO:sockeye.training:__call__] E=10 B=41700	s/sec=2185.21 tok/sec=108330.19 u/sec=24.43	ppl=7.125304 
[2023-12-17:11:50:54:INFO:sockeye.training:__call__] E=10 B=41750	s/sec=2064.81 tok/sec=108702.50 u/sec=24.49	ppl=7.134897 
[2023-12-17:11:50:56:INFO:sockeye.training:__call__] E=10 B=41800	s/sec=2087.79 tok/sec=108421.71 u/sec=24.85	ppl=7.143447 
[2023-12-17:11:50:58:INFO:sockeye.training:__call__] E=10 B=41850	s/sec=2206.53 tok/sec=106200.68 u/sec=24.94	ppl=7.152253 
[2023-12-17:11:51:00:INFO:sockeye.training:__call__] E=10 B=41900	s/sec=2265.03 tok/sec=107507.26 u/sec=24.97	ppl=7.165264 
[2023-12-17:11:51:02:INFO:sockeye.training:__call__] E=10 B=41950	s/sec=2295.68 tok/sec=108387.88 u/sec=24.74	ppl=7.175443 
[2023-12-17:11:51:04:INFO:sockeye.training:__call__] E=10 B=42000	s/sec=2223.96 tok/sec=105942.93 u/sec=25.23	ppl=7.183586 
[2023-12-17:11:51:06:INFO:sockeye.training:__call__] E=10 B=42050	s/sec=2148.39 tok/sec=106359.71 u/sec=25.29	ppl=7.192649 
[2023-12-17:11:51:08:INFO:sockeye.training:__call__] E=10 B=42100	s/sec=2075.54 tok/sec=107817.39 u/sec=24.71	ppl=7.200606 
[2023-12-17:11:51:10:INFO:sockeye.training:__call__] E=10 B=42150	s/sec=2206.48 tok/sec=110051.35 u/sec=24.32	ppl=7.209103 
[2023-12-17:11:51:12:INFO:sockeye.training:__call__] E=10 B=42200	s/sec=2106.05 tok/sec=108516.43 u/sec=24.38	ppl=7.216821 
[2023-12-17:11:51:14:INFO:sockeye.training:__call__] E=10 B=42250	s/sec=2131.02 tok/sec=106606.10 u/sec=24.62	ppl=7.224222 
[2023-12-17:11:51:16:INFO:sockeye.training:__call__] E=10 B=42300	s/sec=2171.41 tok/sec=107394.89 u/sec=24.99	ppl=7.228435 
[2023-12-17:11:51:18:INFO:sockeye.training:__call__] E=10 B=42350	s/sec=2239.72 tok/sec=105198.74 u/sec=25.04	ppl=7.237987 
[2023-12-17:11:51:20:INFO:sockeye.training:__call__] E=10 B=42400	s/sec=2406.51 tok/sec=105210.28 u/sec=25.15	ppl=7.250051 
[2023-12-17:11:51:22:INFO:sockeye.training:__call__] E=10 B=42450	s/sec=2077.85 tok/sec=107877.87 u/sec=24.18	ppl=7.256040 
[2023-12-17:11:51:24:INFO:sockeye.training:__call__] E=10 B=42500	s/sec=2129.55 tok/sec=106617.79 u/sec=24.38	ppl=7.263198 
[2023-12-17:11:51:26:INFO:sockeye.training:__call__] E=10 B=42550	s/sec=2146.50 tok/sec=108787.72 u/sec=24.84	ppl=7.269466 
[2023-12-17:11:51:28:INFO:sockeye.training:__call__] E=10 B=42600	s/sec=2077.69 tok/sec=108134.85 u/sec=24.78	ppl=7.275908 
[2023-12-17:11:51:30:INFO:sockeye.training:__call__] E=10 B=42650	s/sec=2295.46 tok/sec=103141.77 u/sec=25.35	ppl=7.284028 
[2023-12-17:11:51:32:INFO:sockeye.training:__call__] E=10 B=42700	s/sec=2085.96 tok/sec=110642.67 u/sec=24.60	ppl=7.289186 
[2023-12-17:11:51:34:INFO:sockeye.training:__call__] E=10 B=42750	s/sec=2308.99 tok/sec=107763.29 u/sec=24.97	ppl=7.298048 
[2023-12-17:11:51:36:INFO:sockeye.training:__call__] E=10 B=42800	s/sec=2489.68 tok/sec=103157.78 u/sec=25.59	ppl=7.309799 
[2023-12-17:11:51:38:INFO:sockeye.training:__call__] E=10 B=42850	s/sec=2394.24 tok/sec=105672.38 u/sec=25.45	ppl=7.318133 
[2023-12-17:11:51:40:INFO:sockeye.training:__call__] E=10 B=42900	s/sec=2139.10 tok/sec=107225.57 u/sec=24.18	ppl=7.328284 
[2023-12-17:11:51:42:INFO:sockeye.training:__call__] E=10 B=42950	s/sec=2371.67 tok/sec=104288.86 u/sec=25.30	ppl=7.336873 
[2023-12-17:11:51:44:INFO:sockeye.training:__call__] E=10 B=43000	s/sec=2525.32 tok/sec=100979.70 u/sec=25.83	ppl=7.347680 
[2023-12-17:11:51:46:INFO:sockeye.training:__call__] E=10 B=43050	s/sec=2175.00 tok/sec=104790.43 u/sec=25.41	ppl=7.355013 
[2023-12-17:11:51:48:INFO:sockeye.training:__call__] E=10 B=43100	s/sec=2401.35 tok/sec=104231.38 u/sec=25.35	ppl=7.361701 
[2023-12-17:11:51:50:INFO:sockeye.training:__call__] E=10 B=43150	s/sec=2399.74 tok/sec=105140.73 u/sec=25.00	ppl=7.370382 
[2023-12-17:11:51:52:INFO:sockeye.training:__call__] E=10 B=43200	s/sec=2053.14 tok/sec=114769.94 u/sec=22.96	ppl=7.378951 
[2023-12-17:11:51:54:INFO:sockeye.training:__call__] E=10 B=43250	s/sec=1835.19 tok/sec=109634.96 u/sec=24.83	ppl=7.382230 
[2023-12-17:11:51:56:INFO:sockeye.training:__call__] E=10 B=43300	s/sec=2203.80 tok/sec=105132.94 u/sec=25.37	ppl=7.387944 
[2023-12-17:11:51:58:INFO:sockeye.training:__call__] E=10 B=43350	s/sec=2346.95 tok/sec=106011.77 u/sec=24.95	ppl=7.394608 
[2023-12-17:11:52:00:INFO:sockeye.training:__call__] E=10 B=43400	s/sec=2363.16 tok/sec=104906.40 u/sec=25.42	ppl=7.404054 
[2023-12-17:11:52:02:INFO:sockeye.training:__call__] E=10 B=43450	s/sec=2259.91 tok/sec=107751.32 u/sec=24.61	ppl=7.413185 
[2023-12-17:11:52:04:INFO:sockeye.training:__call__] E=10 B=43500	s/sec=2266.06 tok/sec=105282.17 u/sec=25.47	ppl=7.419161 
[2023-12-17:11:52:06:INFO:sockeye.training:__call__] E=10 B=43550	s/sec=2169.82 tok/sec=107622.86 u/sec=24.43	ppl=7.427068 
[2023-12-17:11:52:08:INFO:sockeye.training:__call__] E=10 B=43600	s/sec=2204.44 tok/sec=107018.63 u/sec=24.78	ppl=7.434397 
[2023-12-17:11:52:10:INFO:sockeye.training:__call__] E=10 B=43650	s/sec=2550.10 tok/sec=103173.60 u/sec=25.38	ppl=7.443040 
[2023-12-17:11:52:12:INFO:sockeye.training:__call__] E=10 B=43700	s/sec=1874.46 tok/sec=110754.30 u/sec=24.56	ppl=7.447242 
[2023-12-17:11:52:14:INFO:sockeye.training:__call__] E=10 B=43750	s/sec=2347.98 tok/sec=102165.88 u/sec=25.57	ppl=7.456787 
[2023-12-17:11:52:16:INFO:sockeye.training:__call__] E=10 B=43800	s/sec=2297.82 tok/sec=105523.39 u/sec=25.02	ppl=7.462837 
[2023-12-17:11:52:18:INFO:sockeye.training:__call__] E=10 B=43850	s/sec=1784.26 tok/sec=112369.75 u/sec=24.14	ppl=7.467063 
[2023-12-17:11:52:20:INFO:sockeye.training:__call__] E=10 B=43900	s/sec=2145.70 tok/sec=108058.78 u/sec=24.43	ppl=7.474371 
[2023-12-17:11:52:22:INFO:sockeye.training:__call__] E=11 B=43950	s/sec=2096.90 tok/sec=101106.15 u/sec=23.70	ppl=7.478648 
[2023-12-17:11:52:24:INFO:sockeye.training:__call__] E=11 B=44000	s/sec=2210.24 tok/sec=105898.46 u/sec=25.12	ppl=7.466368 
[2023-12-17:11:52:24:INFO:sockeye.training:_create_checkpoint] Checkpoint [11]	Updates=44000 Epoch=11 Samples=3921272 Time-cost=160.749 Updates/sec=24.884
[2023-12-17:11:52:24:INFO:sockeye.training:_create_checkpoint] Checkpoint [11]	Train-ppl=7.466368
[2023-12-17:11:52:27:INFO:sockeye.training:_evaluate] Checkpoint [11]	Validation-ppl=31.358932	Validation-bleu=0.159950	Validation-chrf=30.898494	Validation-rouge1=0.165754	Validation-rouge2=0.077838	Validation-rougel=0.158876	Validation-length-ratio-mse=1.300733	Validation-ter=97.234833	Validation-avg-sec-per-sent=0.003345	Validation-decode-walltime=1.672364
[2023-12-17:11:52:27:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 6 checkpoints, best so far: 31.983236
[2023-12-17:11:52:27:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [11]	Learning-rate=0.000200
[2023-12-17:11:52:27:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:52:27:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=6, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:52:28:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00011"
[2023-12-17:11:52:30:INFO:sockeye.training:__call__] E=11 B=44050	s/sec=787.44 tok/sec=43343.75 u/sec=9.34	ppl=6.596199 
[2023-12-17:11:52:32:INFO:sockeye.training:__call__] E=11 B=44100	s/sec=2244.31 tok/sec=105004.66 u/sec=25.32	ppl=6.575996 
[2023-12-17:11:52:34:INFO:sockeye.training:__call__] E=11 B=44150	s/sec=2482.25 tok/sec=103177.29 u/sec=25.31	ppl=6.622721 
[2023-12-17:11:52:36:INFO:sockeye.training:__call__] E=11 B=44200	s/sec=1995.54 tok/sec=110228.07 u/sec=24.26	ppl=6.602250 
[2023-12-17:11:52:38:INFO:sockeye.training:__call__] E=11 B=44250	s/sec=2274.70 tok/sec=105184.54 u/sec=25.21	ppl=6.597226 
[2023-12-17:11:52:40:INFO:sockeye.training:__call__] E=11 B=44300	s/sec=2328.13 tok/sec=105987.59 u/sec=24.87	ppl=6.620737 
[2023-12-17:11:52:42:INFO:sockeye.training:__call__] E=11 B=44350	s/sec=2264.14 tok/sec=105971.20 u/sec=25.18	ppl=6.631284 
[2023-12-17:11:52:44:INFO:sockeye.training:__call__] E=11 B=44400	s/sec=2237.56 tok/sec=105474.43 u/sec=25.11	ppl=6.637561 
[2023-12-17:11:52:46:INFO:sockeye.training:__call__] E=11 B=44450	s/sec=2341.67 tok/sec=105975.43 u/sec=24.85	ppl=6.649298 
[2023-12-17:11:52:48:INFO:sockeye.training:__call__] E=11 B=44500	s/sec=2103.79 tok/sec=107810.13 u/sec=25.05	ppl=6.659105 
[2023-12-17:11:52:50:INFO:sockeye.training:__call__] E=11 B=44550	s/sec=2227.48 tok/sec=108362.00 u/sec=24.51	ppl=6.674955 
[2023-12-17:11:52:52:INFO:sockeye.training:__call__] E=11 B=44600	s/sec=2153.35 tok/sec=106882.20 u/sec=24.79	ppl=6.696619 
[2023-12-17:11:52:54:INFO:sockeye.training:__call__] E=11 B=44650	s/sec=1966.04 tok/sec=111325.06 u/sec=24.58	ppl=6.699105 
[2023-12-17:11:52:56:INFO:sockeye.training:__call__] E=11 B=44700	s/sec=2215.79 tok/sec=107736.22 u/sec=25.04	ppl=6.706164 
[2023-12-17:11:52:58:INFO:sockeye.training:__call__] E=11 B=44750	s/sec=2391.01 tok/sec=105739.45 u/sec=24.58	ppl=6.722795 
[2023-12-17:11:53:00:INFO:sockeye.training:__call__] E=11 B=44800	s/sec=2008.53 tok/sec=108803.32 u/sec=24.33	ppl=6.725531 
[2023-12-17:11:53:02:INFO:sockeye.training:__call__] E=11 B=44850	s/sec=2066.44 tok/sec=108563.39 u/sec=24.74	ppl=6.734768 
[2023-12-17:11:53:04:INFO:sockeye.training:__call__] E=11 B=44900	s/sec=2554.94 tok/sec=100547.68 u/sec=26.31	ppl=6.752349 
[2023-12-17:11:53:06:INFO:sockeye.training:__call__] E=11 B=44950	s/sec=2224.97 tok/sec=105788.72 u/sec=24.66	ppl=6.764498 
[2023-12-17:11:53:08:INFO:sockeye.training:__call__] E=11 B=45000	s/sec=2179.91 tok/sec=103953.71 u/sec=25.37	ppl=6.777333 
[2023-12-17:11:53:10:INFO:sockeye.training:__call__] E=11 B=45050	s/sec=2254.45 tok/sec=104212.95 u/sec=25.21	ppl=6.787883 
[2023-12-17:11:53:12:INFO:sockeye.training:__call__] E=11 B=45100	s/sec=2236.32 tok/sec=106146.55 u/sec=24.61	ppl=6.802996 
[2023-12-17:11:53:14:INFO:sockeye.training:__call__] E=11 B=45150	s/sec=2261.19 tok/sec=103597.45 u/sec=25.56	ppl=6.812867 
[2023-12-17:11:53:16:INFO:sockeye.training:__call__] E=11 B=45200	s/sec=2177.90 tok/sec=108170.21 u/sec=24.88	ppl=6.821587 
[2023-12-17:11:53:18:INFO:sockeye.training:__call__] E=11 B=45250	s/sec=1883.10 tok/sec=112955.16 u/sec=24.32	ppl=6.824660 
[2023-12-17:11:53:20:INFO:sockeye.training:__call__] E=11 B=45300	s/sec=2269.35 tok/sec=104065.85 u/sec=25.33	ppl=6.838150 
[2023-12-17:11:53:22:INFO:sockeye.training:__call__] E=11 B=45350	s/sec=2097.39 tok/sec=108536.86 u/sec=24.97	ppl=6.848868 
[2023-12-17:11:53:24:INFO:sockeye.training:__call__] E=11 B=45400	s/sec=2018.93 tok/sec=105750.52 u/sec=24.94	ppl=6.856189 
[2023-12-17:11:53:26:INFO:sockeye.training:__call__] E=11 B=45450	s/sec=2125.67 tok/sec=107700.44 u/sec=24.88	ppl=6.865323 
[2023-12-17:11:53:28:INFO:sockeye.training:__call__] E=11 B=45500	s/sec=2398.02 tok/sec=103065.73 u/sec=25.49	ppl=6.876924 
[2023-12-17:11:53:30:INFO:sockeye.training:__call__] E=11 B=45550	s/sec=2477.19 tok/sec=101696.77 u/sec=25.80	ppl=6.893636 
[2023-12-17:11:53:32:INFO:sockeye.training:__call__] E=11 B=45600	s/sec=2317.13 tok/sec=104597.39 u/sec=25.50	ppl=6.907153 
[2023-12-17:11:53:34:INFO:sockeye.training:__call__] E=11 B=45650	s/sec=2157.02 tok/sec=109715.49 u/sec=24.38	ppl=6.914262 
[2023-12-17:11:53:36:INFO:sockeye.training:__call__] E=11 B=45700	s/sec=2292.51 tok/sec=106221.12 u/sec=24.66	ppl=6.923260 
[2023-12-17:11:53:38:INFO:sockeye.training:__call__] E=11 B=45750	s/sec=2023.85 tok/sec=108859.11 u/sec=24.80	ppl=6.933837 
[2023-12-17:11:53:40:INFO:sockeye.training:__call__] E=11 B=45800	s/sec=2280.61 tok/sec=110380.01 u/sec=24.53	ppl=6.939693 
[2023-12-17:11:53:42:INFO:sockeye.training:__call__] E=11 B=45850	s/sec=2134.06 tok/sec=106814.12 u/sec=24.79	ppl=6.948225 
[2023-12-17:11:53:44:INFO:sockeye.training:__call__] E=11 B=45900	s/sec=2168.13 tok/sec=106439.96 u/sec=24.68	ppl=6.956051 
[2023-12-17:11:53:46:INFO:sockeye.training:__call__] E=11 B=45950	s/sec=2340.20 tok/sec=103789.86 u/sec=25.66	ppl=6.967077 
[2023-12-17:11:53:48:INFO:sockeye.training:__call__] E=11 B=46000	s/sec=2261.65 tok/sec=102615.58 u/sec=25.65	ppl=6.974741 
[2023-12-17:11:53:50:INFO:sockeye.training:__call__] E=11 B=46050	s/sec=2418.81 tok/sec=104551.23 u/sec=25.49	ppl=6.981922 
[2023-12-17:11:53:52:INFO:sockeye.training:__call__] E=11 B=46100	s/sec=2423.33 tok/sec=103230.71 u/sec=25.76	ppl=6.992465 
[2023-12-17:11:53:54:INFO:sockeye.training:__call__] E=11 B=46150	s/sec=1962.09 tok/sec=109156.55 u/sec=24.48	ppl=6.998839 
[2023-12-17:11:53:56:INFO:sockeye.training:__call__] E=11 B=46200	s/sec=2388.62 tok/sec=101717.46 u/sec=25.65	ppl=7.008542 
[2023-12-17:11:53:58:INFO:sockeye.training:__call__] E=11 B=46250	s/sec=2205.77 tok/sec=107568.39 u/sec=24.93	ppl=7.013438 
[2023-12-17:11:54:00:INFO:sockeye.training:__call__] E=11 B=46300	s/sec=2250.29 tok/sec=106890.78 u/sec=25.07	ppl=7.022492 
[2023-12-17:11:54:02:INFO:sockeye.training:__call__] E=11 B=46350	s/sec=2048.99 tok/sec=109916.57 u/sec=24.49	ppl=7.031183 
[2023-12-17:11:54:04:INFO:sockeye.training:__call__] E=11 B=46400	s/sec=2062.41 tok/sec=109338.77 u/sec=24.05	ppl=7.035314 
[2023-12-17:11:54:06:INFO:sockeye.training:__call__] E=11 B=46450	s/sec=2053.54 tok/sec=108670.07 u/sec=24.97	ppl=7.041917 
[2023-12-17:11:54:08:INFO:sockeye.training:__call__] E=11 B=46500	s/sec=2098.45 tok/sec=106311.13 u/sec=24.65	ppl=7.046487 
[2023-12-17:11:54:10:INFO:sockeye.training:__call__] E=11 B=46550	s/sec=2224.77 tok/sec=108412.48 u/sec=24.39	ppl=7.054280 
[2023-12-17:11:54:12:INFO:sockeye.training:__call__] E=11 B=46600	s/sec=2501.82 tok/sec=105554.89 u/sec=25.34	ppl=7.065062 
[2023-12-17:11:54:14:INFO:sockeye.training:__call__] E=11 B=46650	s/sec=2449.26 tok/sec=106099.55 u/sec=25.30	ppl=7.076026 
[2023-12-17:11:54:16:INFO:sockeye.training:__call__] E=11 B=46700	s/sec=2331.24 tok/sec=104258.07 u/sec=25.30	ppl=7.084515 
[2023-12-17:11:54:18:INFO:sockeye.training:__call__] E=11 B=46750	s/sec=2111.38 tok/sec=108109.12 u/sec=24.80	ppl=7.090173 
[2023-12-17:11:54:20:INFO:sockeye.training:__call__] E=11 B=46800	s/sec=2203.37 tok/sec=106312.70 u/sec=25.31	ppl=7.097801 
[2023-12-17:11:54:22:INFO:sockeye.training:__call__] E=11 B=46850	s/sec=2638.10 tok/sec=101177.82 u/sec=25.72	ppl=7.109153 
[2023-12-17:11:54:24:INFO:sockeye.training:__call__] E=11 B=46900	s/sec=2293.44 tok/sec=109329.76 u/sec=24.59	ppl=7.115623 
[2023-12-17:11:54:26:INFO:sockeye.training:__call__] E=11 B=46950	s/sec=2347.57 tok/sec=106275.80 u/sec=24.66	ppl=7.125034 
[2023-12-17:11:54:28:INFO:sockeye.training:__call__] E=11 B=47000	s/sec=2272.76 tok/sec=111965.12 u/sec=23.87	ppl=7.135330 
[2023-12-17:11:54:30:INFO:sockeye.training:__call__] E=11 B=47050	s/sec=2298.52 tok/sec=103425.18 u/sec=25.56	ppl=7.141774 
[2023-12-17:11:54:32:INFO:sockeye.training:__call__] E=11 B=47100	s/sec=2126.03 tok/sec=110475.77 u/sec=24.38	ppl=7.149675 
[2023-12-17:11:54:34:INFO:sockeye.training:__call__] E=11 B=47150	s/sec=2095.29 tok/sec=109297.74 u/sec=24.34	ppl=7.159687 
[2023-12-17:11:54:36:INFO:sockeye.training:__call__] E=11 B=47200	s/sec=2135.34 tok/sec=106182.29 u/sec=25.04	ppl=7.164829 
[2023-12-17:11:54:38:INFO:sockeye.training:__call__] E=11 B=47250	s/sec=2285.04 tok/sec=104828.76 u/sec=25.28	ppl=7.175232 
[2023-12-17:11:54:40:INFO:sockeye.training:__call__] E=11 B=47300	s/sec=2115.07 tok/sec=108492.52 u/sec=24.53	ppl=7.180509 
[2023-12-17:11:54:42:INFO:sockeye.training:__call__] E=11 B=47350	s/sec=2240.49 tok/sec=106771.24 u/sec=25.14	ppl=7.187401 
[2023-12-17:11:54:44:INFO:sockeye.training:__call__] E=11 B=47400	s/sec=2127.58 tok/sec=107407.41 u/sec=24.72	ppl=7.192464 
[2023-12-17:11:54:46:INFO:sockeye.training:__call__] E=11 B=47450	s/sec=2050.90 tok/sec=109244.79 u/sec=24.60	ppl=7.198408 
[2023-12-17:11:54:48:INFO:sockeye.training:__call__] E=11 B=47500	s/sec=2036.55 tok/sec=109646.13 u/sec=24.38	ppl=7.205921 
[2023-12-17:11:54:50:INFO:sockeye.training:__call__] E=11 B=47550	s/sec=2280.40 tok/sec=105940.80 u/sec=25.45	ppl=7.211672 
[2023-12-17:11:54:52:INFO:sockeye.training:__call__] E=11 B=47600	s/sec=1968.45 tok/sec=111721.44 u/sec=24.22	ppl=7.215685 
[2023-12-17:11:54:54:INFO:sockeye.training:__call__] E=11 B=47650	s/sec=2228.46 tok/sec=109413.11 u/sec=24.83	ppl=7.220136 
[2023-12-17:11:54:56:INFO:sockeye.training:__call__] E=11 B=47700	s/sec=2296.61 tok/sec=104705.80 u/sec=25.27	ppl=7.226021 
[2023-12-17:11:54:58:INFO:sockeye.training:__call__] E=11 B=47750	s/sec=2528.26 tok/sec=102856.51 u/sec=25.45	ppl=7.234587 
[2023-12-17:11:55:00:INFO:sockeye.training:__call__] E=11 B=47800	s/sec=2336.39 tok/sec=106293.72 u/sec=24.58	ppl=7.241369 
[2023-12-17:11:55:02:INFO:sockeye.training:__call__] E=11 B=47850	s/sec=2492.84 tok/sec=101947.65 u/sec=25.67	ppl=7.250633 
[2023-12-17:11:55:04:INFO:sockeye.training:__call__] E=11 B=47900	s/sec=2173.41 tok/sec=108094.17 u/sec=24.34	ppl=7.258478 
[2023-12-17:11:55:06:INFO:sockeye.training:__call__] E=12 B=47950	s/sec=2048.65 tok/sec=102652.72 u/sec=23.76	ppl=7.261173 
[2023-12-17:11:55:08:INFO:sockeye.training:__call__] E=12 B=48000	s/sec=2044.64 tok/sec=109205.75 u/sec=24.77	ppl=7.249319 
[2023-12-17:11:55:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [12]	Updates=48000 Epoch=12 Samples=4277480 Time-cost=160.585 Updates/sec=24.909
[2023-12-17:11:55:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [12]	Train-ppl=7.249319
[2023-12-17:11:55:29:INFO:sockeye.training:_evaluate] Checkpoint [12]	Validation-ppl=33.066800	Validation-bleu=0.159157	Validation-chrf=29.846615	Validation-rouge1=0.145661	Validation-rouge2=0.071502	Validation-rougel=0.138529	Validation-length-ratio-mse=1.286198	Validation-ter=98.060256	Validation-avg-sec-per-sent=0.039493	Validation-decode-walltime=19.746372
[2023-12-17:11:55:29:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 7 checkpoints, best so far: 31.983236
[2023-12-17:11:55:29:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [12]	Learning-rate=0.000200
[2023-12-17:11:55:29:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:55:29:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=7, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:55:29:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00012"
[2023-12-17:11:55:31:INFO:sockeye.training:__call__] E=12 B=48050	s/sec=187.20 tok/sec=8897.84 u/sec=2.15	ppl=6.426198 
[2023-12-17:11:55:33:INFO:sockeye.training:__call__] E=12 B=48100	s/sec=2119.12 tok/sec=109438.02 u/sec=24.62	ppl=6.454798 
[2023-12-17:11:55:35:INFO:sockeye.training:__call__] E=12 B=48150	s/sec=2248.92 tok/sec=108333.91 u/sec=25.10	ppl=6.435897 
[2023-12-17:11:55:37:INFO:sockeye.training:__call__] E=12 B=48200	s/sec=2376.57 tok/sec=103574.25 u/sec=25.48	ppl=6.435910 
[2023-12-17:11:55:39:INFO:sockeye.training:__call__] E=12 B=48250	s/sec=2078.96 tok/sec=107551.61 u/sec=24.75	ppl=6.436725 
[2023-12-17:11:55:41:INFO:sockeye.training:__call__] E=12 B=48300	s/sec=2087.50 tok/sec=110539.43 u/sec=24.48	ppl=6.435905 
[2023-12-17:11:55:43:INFO:sockeye.training:__call__] E=12 B=48350	s/sec=2362.76 tok/sec=104172.97 u/sec=25.42	ppl=6.456136 
[2023-12-17:11:55:45:INFO:sockeye.training:__call__] E=12 B=48400	s/sec=2246.93 tok/sec=106600.66 u/sec=25.08	ppl=6.472729 
[2023-12-17:11:55:48:INFO:sockeye.training:__call__] E=12 B=48450	s/sec=2013.69 tok/sec=110929.25 u/sec=23.93	ppl=6.469757 
[2023-12-17:11:55:50:INFO:sockeye.training:__call__] E=12 B=48500	s/sec=2089.42 tok/sec=108571.04 u/sec=24.69	ppl=6.490189 
[2023-12-17:11:55:52:INFO:sockeye.training:__call__] E=12 B=48550	s/sec=2256.98 tok/sec=106510.24 u/sec=25.01	ppl=6.511429 
[2023-12-17:11:55:54:INFO:sockeye.training:__call__] E=12 B=48600	s/sec=2181.74 tok/sec=109158.15 u/sec=24.75	ppl=6.516178 
[2023-12-17:11:55:56:INFO:sockeye.training:__call__] E=12 B=48650	s/sec=2245.82 tok/sec=104388.15 u/sec=25.38	ppl=6.528555 
[2023-12-17:11:55:58:INFO:sockeye.training:__call__] E=12 B=48700	s/sec=2208.59 tok/sec=106592.44 u/sec=25.19	ppl=6.536831 
[2023-12-17:11:55:59:INFO:sockeye.training:__call__] E=12 B=48750	s/sec=2413.68 tok/sec=103966.31 u/sec=25.88	ppl=6.558031 
[2023-12-17:11:56:01:INFO:sockeye.training:__call__] E=12 B=48800	s/sec=2100.63 tok/sec=106718.43 u/sec=24.87	ppl=6.568260 
[2023-12-17:11:56:03:INFO:sockeye.training:__call__] E=12 B=48850	s/sec=2173.56 tok/sec=106910.55 u/sec=24.88	ppl=6.580139 
[2023-12-17:11:56:06:INFO:sockeye.training:__call__] E=12 B=48900	s/sec=2122.97 tok/sec=109939.42 u/sec=24.53	ppl=6.588230 
[2023-12-17:11:56:08:INFO:sockeye.training:__call__] E=12 B=48950	s/sec=2006.20 tok/sec=108342.91 u/sec=24.63	ppl=6.598767 
[2023-12-17:11:56:10:INFO:sockeye.training:__call__] E=12 B=49000	s/sec=2324.12 tok/sec=107575.63 u/sec=24.50	ppl=6.610129 
[2023-12-17:11:56:12:INFO:sockeye.training:__call__] E=12 B=49050	s/sec=1990.24 tok/sec=107657.99 u/sec=25.13	ppl=6.613692 
[2023-12-17:11:56:14:INFO:sockeye.training:__call__] E=12 B=49100	s/sec=2371.55 tok/sec=104187.61 u/sec=25.12	ppl=6.629302 
[2023-12-17:11:56:16:INFO:sockeye.training:__call__] E=12 B=49150	s/sec=1876.46 tok/sec=114407.82 u/sec=23.50	ppl=6.630315 
[2023-12-17:11:56:18:INFO:sockeye.training:__call__] E=12 B=49200	s/sec=2240.96 tok/sec=111963.77 u/sec=23.86	ppl=6.641323 
[2023-12-17:11:56:20:INFO:sockeye.training:__call__] E=12 B=49250	s/sec=2056.58 tok/sec=108752.81 u/sec=24.81	ppl=6.647507 
[2023-12-17:11:56:22:INFO:sockeye.training:__call__] E=12 B=49300	s/sec=2198.64 tok/sec=107721.15 u/sec=25.12	ppl=6.659405 
[2023-12-17:11:56:24:INFO:sockeye.training:__call__] E=12 B=49350	s/sec=2177.21 tok/sec=107254.77 u/sec=24.97	ppl=6.666703 
[2023-12-17:11:56:26:INFO:sockeye.training:__call__] E=12 B=49400	s/sec=2126.93 tok/sec=109091.75 u/sec=24.30	ppl=6.674925 
[2023-12-17:11:56:28:INFO:sockeye.training:__call__] E=12 B=49450	s/sec=2281.88 tok/sec=106167.66 u/sec=25.02	ppl=6.683646 
[2023-12-17:11:56:30:INFO:sockeye.training:__call__] E=12 B=49500	s/sec=2239.57 tok/sec=103967.64 u/sec=25.08	ppl=6.693209 
[2023-12-17:11:56:32:INFO:sockeye.training:__call__] E=12 B=49550	s/sec=2120.48 tok/sec=106875.43 u/sec=25.34	ppl=6.702425 
[2023-12-17:11:56:34:INFO:sockeye.training:__call__] E=12 B=49600	s/sec=2422.04 tok/sec=102760.38 u/sec=25.66	ppl=6.714384 
[2023-12-17:11:56:36:INFO:sockeye.training:__call__] E=12 B=49650	s/sec=2546.20 tok/sec=104624.18 u/sec=25.22	ppl=6.726254 
[2023-12-17:11:56:38:INFO:sockeye.training:__call__] E=12 B=49700	s/sec=2267.87 tok/sec=104840.60 u/sec=25.31	ppl=6.733634 
[2023-12-17:11:56:40:INFO:sockeye.training:__call__] E=12 B=49750	s/sec=2392.56 tok/sec=104025.67 u/sec=25.30	ppl=6.749544 
[2023-12-17:11:56:42:INFO:sockeye.training:__call__] E=12 B=49800	s/sec=2292.35 tok/sec=104965.08 u/sec=24.74	ppl=6.760886 
[2023-12-17:11:56:44:INFO:sockeye.training:__call__] E=12 B=49850	s/sec=2399.28 tok/sec=105552.42 u/sec=25.12	ppl=6.770585 
[2023-12-17:11:56:46:INFO:sockeye.training:__call__] E=12 B=49900	s/sec=2299.22 tok/sec=105364.12 u/sec=25.52	ppl=6.780839 
[2023-12-17:11:56:48:INFO:sockeye.training:__call__] E=12 B=49950	s/sec=2401.64 tok/sec=104036.64 u/sec=24.81	ppl=6.791709 
[2023-12-17:11:56:50:INFO:sockeye.training:__call__] E=12 B=50000	s/sec=1988.48 tok/sec=108376.01 u/sec=24.76	ppl=6.796775 
[2023-12-17:11:56:52:INFO:sockeye.training:__call__] E=12 B=50050	s/sec=2621.11 tok/sec=98881.70 u/sec=25.88	ppl=6.809985 
[2023-12-17:11:56:54:INFO:sockeye.training:__call__] E=12 B=50100	s/sec=2421.09 tok/sec=101308.20 u/sec=26.22	ppl=6.821791 
[2023-12-17:11:56:56:INFO:sockeye.training:__call__] E=12 B=50150	s/sec=2310.29 tok/sec=106122.33 u/sec=24.85	ppl=6.825325 
[2023-12-17:11:56:58:INFO:sockeye.training:__call__] E=12 B=50200	s/sec=2055.00 tok/sec=107700.75 u/sec=24.33	ppl=6.834425 
[2023-12-17:11:57:00:INFO:sockeye.training:__call__] E=12 B=50250	s/sec=2288.31 tok/sec=107146.85 u/sec=24.74	ppl=6.842064 
[2023-12-17:11:57:02:INFO:sockeye.training:__call__] E=12 B=50300	s/sec=2219.24 tok/sec=106555.35 u/sec=24.99	ppl=6.850033 
[2023-12-17:11:57:04:INFO:sockeye.training:__call__] E=12 B=50350	s/sec=2109.10 tok/sec=107637.88 u/sec=24.28	ppl=6.859065 
[2023-12-17:11:57:06:INFO:sockeye.training:__call__] E=12 B=50400	s/sec=2279.10 tok/sec=105081.90 u/sec=25.35	ppl=6.866874 
[2023-12-17:11:57:08:INFO:sockeye.training:__call__] E=12 B=50450	s/sec=2149.34 tok/sec=106376.04 u/sec=25.06	ppl=6.873471 
[2023-12-17:11:57:10:INFO:sockeye.training:__call__] E=12 B=50500	s/sec=2181.52 tok/sec=105872.28 u/sec=25.16	ppl=6.881956 
[2023-12-17:11:57:12:INFO:sockeye.training:__call__] E=12 B=50550	s/sec=2513.93 tok/sec=104119.64 u/sec=25.30	ppl=6.892062 
[2023-12-17:11:57:14:INFO:sockeye.training:__call__] E=12 B=50600	s/sec=1934.95 tok/sec=111409.45 u/sec=24.33	ppl=6.898520 
[2023-12-17:11:57:16:INFO:sockeye.training:__call__] E=12 B=50650	s/sec=2164.21 tok/sec=108575.86 u/sec=24.28	ppl=6.905523 
[2023-12-17:11:57:18:INFO:sockeye.training:__call__] E=12 B=50700	s/sec=2359.04 tok/sec=103233.20 u/sec=25.20	ppl=6.915770 
[2023-12-17:11:57:20:INFO:sockeye.training:__call__] E=12 B=50750	s/sec=2077.66 tok/sec=109428.38 u/sec=24.14	ppl=6.921607 
[2023-12-17:11:57:22:INFO:sockeye.training:__call__] E=12 B=50800	s/sec=2346.59 tok/sec=102743.76 u/sec=25.51	ppl=6.927190 
[2023-12-17:11:57:24:INFO:sockeye.training:__call__] E=12 B=50850	s/sec=2349.12 tok/sec=103526.91 u/sec=25.85	ppl=6.933314 
[2023-12-17:11:57:26:INFO:sockeye.training:__call__] E=12 B=50900	s/sec=2069.88 tok/sec=108287.78 u/sec=24.93	ppl=6.939084 
[2023-12-17:11:57:28:INFO:sockeye.training:__call__] E=12 B=50950	s/sec=2419.61 tok/sec=103978.54 u/sec=25.20	ppl=6.946972 
[2023-12-17:11:57:30:INFO:sockeye.training:__call__] E=12 B=51000	s/sec=2120.73 tok/sec=108486.27 u/sec=24.77	ppl=6.952618 
[2023-12-17:11:57:32:INFO:sockeye.training:__call__] E=12 B=51050	s/sec=1998.05 tok/sec=108580.43 u/sec=24.63	ppl=6.960038 
[2023-12-17:11:57:34:INFO:sockeye.training:__call__] E=12 B=51100	s/sec=2125.08 tok/sec=109639.81 u/sec=24.55	ppl=6.966718 
[2023-12-17:11:57:36:INFO:sockeye.training:__call__] E=12 B=51150	s/sec=2272.70 tok/sec=106733.39 u/sec=24.88	ppl=6.974155 
[2023-12-17:11:57:38:INFO:sockeye.training:__call__] E=12 B=51200	s/sec=2111.77 tok/sec=106606.79 u/sec=24.67	ppl=6.980222 
[2023-12-17:11:57:40:INFO:sockeye.training:__call__] E=12 B=51250	s/sec=1914.46 tok/sec=110758.41 u/sec=24.32	ppl=6.984163 
[2023-12-17:11:57:42:INFO:sockeye.training:__call__] E=12 B=51300	s/sec=2303.86 tok/sec=106574.55 u/sec=24.87	ppl=6.990335 
[2023-12-17:11:57:44:INFO:sockeye.training:__call__] E=12 B=51350	s/sec=2401.88 tok/sec=104651.01 u/sec=25.19	ppl=6.997571 
[2023-12-17:11:57:46:INFO:sockeye.training:__call__] E=12 B=51400	s/sec=2174.28 tok/sec=105967.33 u/sec=25.03	ppl=7.004640 
[2023-12-17:11:57:48:INFO:sockeye.training:__call__] E=12 B=51450	s/sec=2049.84 tok/sec=106265.04 u/sec=25.52	ppl=7.010402 
[2023-12-17:11:57:50:INFO:sockeye.training:__call__] E=12 B=51500	s/sec=2331.24 tok/sec=104023.06 u/sec=25.43	ppl=7.017109 
[2023-12-17:11:57:52:INFO:sockeye.training:__call__] E=12 B=51550	s/sec=2391.87 tok/sec=106308.78 u/sec=24.87	ppl=7.024825 
[2023-12-17:11:57:54:INFO:sockeye.training:__call__] E=12 B=51600	s/sec=2308.16 tok/sec=106087.42 u/sec=25.00	ppl=7.030282 
[2023-12-17:11:57:56:INFO:sockeye.training:__call__] E=12 B=51650	s/sec=2270.90 tok/sec=108177.20 u/sec=24.81	ppl=7.037062 
[2023-12-17:11:57:58:INFO:sockeye.training:__call__] E=12 B=51700	s/sec=2356.27 tok/sec=105663.43 u/sec=25.05	ppl=7.044281 
[2023-12-17:11:58:00:INFO:sockeye.training:__call__] E=12 B=51750	s/sec=2296.79 tok/sec=105723.80 u/sec=24.88	ppl=7.049364 
[2023-12-17:11:58:02:INFO:sockeye.training:__call__] E=12 B=51800	s/sec=2419.38 tok/sec=104952.98 u/sec=24.67	ppl=7.056394 
[2023-12-17:11:58:04:INFO:sockeye.training:__call__] E=12 B=51850	s/sec=2249.81 tok/sec=107832.29 u/sec=24.80	ppl=7.062193 
[2023-12-17:11:58:06:INFO:sockeye.training:__call__] E=12 B=51900	s/sec=2357.93 tok/sec=103157.19 u/sec=25.67	ppl=7.068610 
[2023-12-17:11:58:08:INFO:sockeye.training:__call__] E=13 B=51950	s/sec=2025.37 tok/sec=104402.15 u/sec=23.49	ppl=7.069562 
[2023-12-17:11:58:10:INFO:sockeye.training:__call__] E=13 B=52000	s/sec=2254.88 tok/sec=105130.78 u/sec=25.26	ppl=7.058896 
[2023-12-17:11:58:10:INFO:sockeye.training:_create_checkpoint] Checkpoint [13]	Updates=52000 Epoch=13 Samples=4634504 Time-cost=160.477 Updates/sec=24.926
[2023-12-17:11:58:10:INFO:sockeye.training:_create_checkpoint] Checkpoint [13]	Train-ppl=7.058896
[2023-12-17:11:58:13:INFO:sockeye.training:_evaluate] Checkpoint [13]	Validation-ppl=33.000284	Validation-bleu=0.155498	Validation-chrf=30.232177	Validation-rouge1=0.158555	Validation-rouge2=0.079668	Validation-rougel=0.151938	Validation-length-ratio-mse=1.332198	Validation-ter=99.793644	Validation-avg-sec-per-sent=0.003355	Validation-decode-walltime=1.677501
[2023-12-17:11:58:13:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 8 checkpoints, best so far: 31.983236
[2023-12-17:11:58:13:INFO:sockeye.lr_scheduler:new_evaluation_result] 8 checkpoints since improvement or rate scaling, lowering learning rate: 2.00e-04 -> 1.80e-04
[2023-12-17:11:58:13:INFO:sockeye.training:_adjust_learning_rate] Loading model parameters and optimizer states from best checkpoint: 5
[2023-12-17:11:58:13:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best" to "cuda:0"
[2023-12-17:11:58:13:INFO:sockeye.training:_load_optimizer_state] Loaded optimizer state from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:11:58:13:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [13]	Learning-rate=0.000200
[2023-12-17:11:58:13:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:11:58:13:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:11:58:14:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00013"
[2023-12-17:11:58:16:INFO:sockeye.training:__call__] E=13 B=52050	s/sec=726.89 tok/sec=38007.32 u/sec=8.60	ppl=8.404862 
[2023-12-17:11:58:18:INFO:sockeye.training:__call__] E=13 B=52100	s/sec=2008.05 tok/sec=99552.15 u/sec=23.11	ppl=8.350213 
[2023-12-17:11:58:20:INFO:sockeye.training:__call__] E=13 B=52150	s/sec=2345.45 tok/sec=96481.33 u/sec=23.64	ppl=8.401378 
[2023-12-17:11:58:22:INFO:sockeye.training:__call__] E=13 B=52200	s/sec=2040.64 tok/sec=101575.49 u/sec=22.30	ppl=8.393537 
[2023-12-17:11:58:24:INFO:sockeye.training:__call__] E=13 B=52250	s/sec=2066.95 tok/sec=102392.57 u/sec=22.78	ppl=8.448532 
[2023-12-17:11:58:27:INFO:sockeye.training:__call__] E=13 B=52300	s/sec=2065.01 tok/sec=97847.48 u/sec=23.13	ppl=8.445333 
[2023-12-17:11:58:29:INFO:sockeye.training:__call__] E=13 B=52350	s/sec=2027.90 tok/sec=97915.24 u/sec=23.69	ppl=8.430912 
[2023-12-17:11:58:31:INFO:sockeye.training:__call__] E=13 B=52400	s/sec=1934.62 tok/sec=100066.58 u/sec=23.16	ppl=8.433464 
[2023-12-17:11:58:33:INFO:sockeye.training:__call__] E=13 B=52450	s/sec=2222.58 tok/sec=95385.25 u/sec=23.66	ppl=8.460779 
[2023-12-17:11:58:35:INFO:sockeye.training:__call__] E=13 B=52500	s/sec=2141.58 tok/sec=97443.59 u/sec=23.36	ppl=8.473897 
[2023-12-17:11:58:37:INFO:sockeye.training:__call__] E=13 B=52550	s/sec=1905.14 tok/sec=99957.05 u/sec=23.17	ppl=8.460962 
[2023-12-17:11:58:40:INFO:sockeye.training:__call__] E=13 B=52600	s/sec=1917.57 tok/sec=101580.08 u/sec=22.78	ppl=8.439509 
[2023-12-17:11:58:42:INFO:sockeye.training:__call__] E=13 B=52650	s/sec=2063.07 tok/sec=101787.77 u/sec=22.70	ppl=8.452924 
[2023-12-17:11:58:44:INFO:sockeye.training:__call__] E=13 B=52700	s/sec=1895.45 tok/sec=104412.35 u/sec=22.56	ppl=8.457822 
[2023-12-17:11:58:46:INFO:sockeye.training:__call__] E=13 B=52750	s/sec=2283.93 tok/sec=95254.98 u/sec=23.79	ppl=8.478810 
[2023-12-17:11:58:48:INFO:sockeye.training:__call__] E=13 B=52800	s/sec=1881.25 tok/sec=98997.56 u/sec=23.19	ppl=8.470455 
[2023-12-17:11:58:50:INFO:sockeye.training:__call__] E=13 B=52850	s/sec=1974.73 tok/sec=100359.12 u/sec=23.16	ppl=8.467060 
[2023-12-17:11:58:52:INFO:sockeye.training:__call__] E=13 B=52900	s/sec=2433.91 tok/sec=91240.33 u/sec=24.38	ppl=8.493150 
[2023-12-17:11:58:55:INFO:sockeye.training:__call__] E=13 B=52950	s/sec=2214.29 tok/sec=95566.65 u/sec=23.66	ppl=8.505083 
[2023-12-17:11:58:57:INFO:sockeye.training:__call__] E=13 B=53000	s/sec=2004.42 tok/sec=109002.22 u/sec=21.31	ppl=8.503180 
[2023-12-17:11:58:59:INFO:sockeye.training:__call__] E=13 B=53050	s/sec=2093.70 tok/sec=98793.47 u/sec=22.96	ppl=8.521538 
[2023-12-17:11:59:01:INFO:sockeye.training:__call__] E=13 B=53100	s/sec=1819.17 tok/sec=100948.78 u/sec=23.30	ppl=8.526166 
[2023-12-17:11:59:03:INFO:sockeye.training:__call__] E=13 B=53150	s/sec=2001.61 tok/sec=101171.15 u/sec=22.87	ppl=8.535924 
[2023-12-17:11:59:06:INFO:sockeye.training:__call__] E=13 B=53200	s/sec=2148.46 tok/sec=98223.99 u/sec=23.35	ppl=8.559269 
[2023-12-17:11:59:08:INFO:sockeye.training:__call__] E=13 B=53250	s/sec=1965.06 tok/sec=100311.68 u/sec=23.39	ppl=8.567698 
[2023-12-17:11:59:10:INFO:sockeye.training:__call__] E=13 B=53300	s/sec=1675.42 tok/sec=106716.59 u/sec=22.14	ppl=8.565260 
[2023-12-17:11:59:12:INFO:sockeye.training:__call__] E=13 B=53350	s/sec=1850.23 tok/sec=102123.77 u/sec=22.81	ppl=8.576345 
[2023-12-17:11:59:14:INFO:sockeye.training:__call__] E=13 B=53400	s/sec=2271.97 tok/sec=97415.05 u/sec=23.32	ppl=8.591455 
[2023-12-17:11:59:17:INFO:sockeye.training:__call__] E=13 B=53450	s/sec=2118.48 tok/sec=98152.82 u/sec=22.63	ppl=8.609430 
[2023-12-17:11:59:19:INFO:sockeye.training:__call__] E=13 B=53500	s/sec=2211.54 tok/sec=94936.43 u/sec=23.75	ppl=8.630853 
[2023-12-17:11:59:21:INFO:sockeye.training:__call__] E=13 B=53550	s/sec=1908.99 tok/sec=101900.32 u/sec=22.73	ppl=8.638923 
[2023-12-17:11:59:23:INFO:sockeye.training:__call__] E=13 B=53600	s/sec=2108.03 tok/sec=96141.52 u/sec=23.74	ppl=8.657415 
[2023-12-17:11:59:25:INFO:sockeye.training:__call__] E=13 B=53650	s/sec=2017.01 tok/sec=98326.53 u/sec=23.13	ppl=8.660521 
[2023-12-17:11:59:27:INFO:sockeye.training:__call__] E=13 B=53700	s/sec=2213.12 tok/sec=96306.73 u/sec=23.56	ppl=8.677801 
[2023-12-17:11:59:29:INFO:sockeye.training:__call__] E=13 B=53750	s/sec=2173.43 tok/sec=99008.37 u/sec=22.95	ppl=8.689920 
[2023-12-17:11:59:32:INFO:sockeye.training:__call__] E=13 B=53800	s/sec=1997.61 tok/sec=99954.40 u/sec=23.04	ppl=8.696036 
[2023-12-17:11:59:34:INFO:sockeye.training:__call__] E=13 B=53850	s/sec=2247.93 tok/sec=94307.66 u/sec=23.49	ppl=8.711733 
[2023-12-17:11:59:36:INFO:sockeye.training:__call__] E=13 B=53900	s/sec=2182.22 tok/sec=96756.21 u/sec=23.56	ppl=8.724537 
[2023-12-17:11:59:38:INFO:sockeye.training:__call__] E=13 B=53950	s/sec=1927.66 tok/sec=101418.82 u/sec=22.77	ppl=8.733685 
[2023-12-17:11:59:40:INFO:sockeye.training:__call__] E=13 B=54000	s/sec=2036.95 tok/sec=98726.71 u/sec=23.27	ppl=8.747541 
[2023-12-17:11:59:42:INFO:sockeye.training:__call__] E=13 B=54050	s/sec=2096.13 tok/sec=100147.79 u/sec=22.78	ppl=8.757429 
[2023-12-17:11:59:45:INFO:sockeye.training:__call__] E=13 B=54100	s/sec=1969.68 tok/sec=103610.15 u/sec=22.34	ppl=8.768757 
[2023-12-17:11:59:47:INFO:sockeye.training:__call__] E=13 B=54150	s/sec=2228.93 tok/sec=96634.83 u/sec=23.65	ppl=8.783008 
[2023-12-17:11:59:49:INFO:sockeye.training:__call__] E=13 B=54200	s/sec=2023.79 tok/sec=99109.71 u/sec=23.29	ppl=8.791716 
[2023-12-17:11:59:51:INFO:sockeye.training:__call__] E=13 B=54250	s/sec=2010.79 tok/sec=100232.67 u/sec=23.40	ppl=8.800845 
[2023-12-17:11:59:53:INFO:sockeye.training:__call__] E=13 B=54300	s/sec=2295.99 tok/sec=94875.44 u/sec=23.96	ppl=8.815299 
[2023-12-17:11:59:55:INFO:sockeye.training:__call__] E=13 B=54350	s/sec=2077.72 tok/sec=98015.10 u/sec=23.11	ppl=8.826372 
[2023-12-17:11:59:57:INFO:sockeye.training:__call__] E=13 B=54400	s/sec=1836.89 tok/sec=102077.24 u/sec=22.82	ppl=8.831585 
[2023-12-17:12:00:00:INFO:sockeye.training:__call__] E=13 B=54450	s/sec=2044.76 tok/sec=97728.83 u/sec=23.41	ppl=8.842119 
[2023-12-17:12:00:02:INFO:sockeye.training:__call__] E=13 B=54500	s/sec=1965.84 tok/sec=100073.58 u/sec=23.01	ppl=8.848059 
[2023-12-17:12:00:04:INFO:sockeye.training:__call__] E=13 B=54550	s/sec=2025.15 tok/sec=99877.29 u/sec=23.44	ppl=8.859884 
[2023-12-17:12:00:06:INFO:sockeye.training:__call__] E=13 B=54600	s/sec=2328.71 tok/sec=95087.07 u/sec=23.67	ppl=8.870886 
[2023-12-17:12:00:08:INFO:sockeye.training:__call__] E=13 B=54650	s/sec=2207.39 tok/sec=99123.65 u/sec=23.30	ppl=8.883427 
[2023-12-17:12:00:10:INFO:sockeye.training:__call__] E=13 B=54700	s/sec=2165.66 tok/sec=96187.44 u/sec=23.79	ppl=8.891843 
[2023-12-17:12:00:12:INFO:sockeye.training:__call__] E=13 B=54750	s/sec=1602.75 tok/sec=102719.44 u/sec=22.46	ppl=8.892550 
[2023-12-17:12:00:15:INFO:sockeye.training:__call__] E=13 B=54800	s/sec=2039.83 tok/sec=98763.02 u/sec=22.93	ppl=8.901786 
[2023-12-17:12:00:17:INFO:sockeye.training:__call__] E=13 B=54850	s/sec=2270.48 tok/sec=96177.55 u/sec=23.65	ppl=8.912370 
[2023-12-17:12:00:19:INFO:sockeye.training:__call__] E=13 B=54900	s/sec=2087.58 tok/sec=98315.00 u/sec=23.42	ppl=8.918003 
[2023-12-17:12:00:21:INFO:sockeye.training:__call__] E=13 B=54950	s/sec=2118.49 tok/sec=98823.49 u/sec=23.07	ppl=8.923288 
[2023-12-17:12:00:23:INFO:sockeye.training:__call__] E=13 B=55000	s/sec=1904.59 tok/sec=102760.10 u/sec=22.89	ppl=8.928414 
[2023-12-17:12:00:25:INFO:sockeye.training:__call__] E=13 B=55050	s/sec=2100.08 tok/sec=95478.92 u/sec=23.91	ppl=8.936910 
[2023-12-17:12:00:28:INFO:sockeye.training:__call__] E=13 B=55100	s/sec=1865.74 tok/sec=102543.60 u/sec=22.60	ppl=8.942789 
[2023-12-17:12:00:30:INFO:sockeye.training:__call__] E=13 B=55150	s/sec=2114.24 tok/sec=98815.87 u/sec=22.90	ppl=8.954341 
[2023-12-17:12:00:32:INFO:sockeye.training:__call__] E=13 B=55200	s/sec=1858.20 tok/sec=100521.13 u/sec=23.23	ppl=8.961992 
[2023-12-17:12:00:34:INFO:sockeye.training:__call__] E=13 B=55250	s/sec=2245.44 tok/sec=98073.37 u/sec=23.63	ppl=8.973448 
[2023-12-17:12:00:36:INFO:sockeye.training:__call__] E=13 B=55300	s/sec=2046.55 tok/sec=100047.12 u/sec=23.21	ppl=8.982648 
[2023-12-17:12:00:38:INFO:sockeye.training:__call__] E=13 B=55350	s/sec=2209.19 tok/sec=95612.75 u/sec=23.68	ppl=8.991393 
[2023-12-17:12:00:40:INFO:sockeye.training:__call__] E=13 B=55400	s/sec=2197.80 tok/sec=97941.97 u/sec=23.32	ppl=8.999314 
[2023-12-17:12:00:43:INFO:sockeye.training:__call__] E=13 B=55450	s/sec=2273.91 tok/sec=97453.41 u/sec=23.61	ppl=9.010148 
[2023-12-17:12:00:45:INFO:sockeye.training:__call__] E=13 B=55500	s/sec=1891.25 tok/sec=102819.95 u/sec=22.78	ppl=9.016816 
[2023-12-17:12:00:47:INFO:sockeye.training:__call__] E=13 B=55550	s/sec=2037.20 tok/sec=99527.45 u/sec=23.07	ppl=9.023608 
[2023-12-17:12:00:49:INFO:sockeye.training:__call__] E=13 B=55600	s/sec=1930.95 tok/sec=101424.63 u/sec=22.99	ppl=9.030694 
[2023-12-17:12:00:51:INFO:sockeye.training:__call__] E=13 B=55650	s/sec=2211.75 tok/sec=97779.49 u/sec=23.31	ppl=9.039072 
[2023-12-17:12:00:53:INFO:sockeye.training:__call__] E=13 B=55700	s/sec=2171.35 tok/sec=96711.62 u/sec=23.48	ppl=9.046591 
[2023-12-17:12:00:56:INFO:sockeye.training:__call__] E=13 B=55750	s/sec=2032.78 tok/sec=100825.69 u/sec=23.10	ppl=9.052814 
[2023-12-17:12:00:58:INFO:sockeye.training:__call__] E=13 B=55800	s/sec=2104.47 tok/sec=102400.45 u/sec=22.56	ppl=9.061085 
[2023-12-17:12:01:00:INFO:sockeye.training:__call__] E=13 B=55850	s/sec=2232.20 tok/sec=96052.05 u/sec=23.49	ppl=9.069080 
[2023-12-17:12:01:02:INFO:sockeye.training:__call__] E=13 B=55900	s/sec=2007.49 tok/sec=97896.36 u/sec=23.54	ppl=9.075221 
[2023-12-17:12:01:04:INFO:sockeye.training:__call__] E=14 B=55950	s/sec=2035.50 tok/sec=95284.25 u/sec=22.24	ppl=9.072330 
[2023-12-17:12:01:06:INFO:sockeye.training:__call__] E=14 B=56000	s/sec=2063.46 tok/sec=99808.38 u/sec=22.91	ppl=9.054902 
[2023-12-17:12:01:06:INFO:sockeye.training:_create_checkpoint] Checkpoint [14]	Updates=56000 Epoch=14 Samples=4991032 Time-cost=172.854 Updates/sec=23.141
[2023-12-17:12:01:06:INFO:sockeye.training:_create_checkpoint] Checkpoint [14]	Train-ppl=9.054902
[2023-12-17:12:01:09:INFO:sockeye.training:_evaluate] Checkpoint [14]	Validation-ppl=27.111713	Validation-bleu=0.144674	Validation-chrf=30.229130	Validation-rouge1=0.157525	Validation-rouge2=0.073042	Validation-rougel=0.152280	Validation-length-ratio-mse=1.289199	Validation-ter=96.491952	Validation-avg-sec-per-sent=0.003481	Validation-decode-walltime=1.740565
[2023-12-17:12:01:09:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 9 checkpoints, best so far: 31.983236
[2023-12-17:12:01:09:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [14]	Learning-rate=0.000180
[2023-12-17:12:01:09:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:01:09:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:01:10:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00014"
[2023-12-17:12:01:12:INFO:sockeye.training:__call__] E=14 B=56050	s/sec=842.52 tok/sec=39004.33 u/sec=9.13	ppl=7.859161 
[2023-12-17:12:01:14:INFO:sockeye.training:__call__] E=14 B=56100	s/sec=2054.10 tok/sec=96813.89 u/sec=23.22	ppl=7.786871 
[2023-12-17:12:01:16:INFO:sockeye.training:__call__] E=14 B=56150	s/sec=2008.83 tok/sec=98455.07 u/sec=23.34	ppl=7.822484 
[2023-12-17:12:01:18:INFO:sockeye.training:__call__] E=14 B=56200	s/sec=1864.77 tok/sec=101959.99 u/sec=22.67	ppl=7.816300 
[2023-12-17:12:01:20:INFO:sockeye.training:__call__] E=14 B=56250	s/sec=2314.80 tok/sec=95698.65 u/sec=23.80	ppl=7.864802 
[2023-12-17:12:01:23:INFO:sockeye.training:__call__] E=14 B=56300	s/sec=2036.59 tok/sec=100820.64 u/sec=23.02	ppl=7.860117 
[2023-12-17:12:01:25:INFO:sockeye.training:__call__] E=14 B=56350	s/sec=1979.93 tok/sec=99617.57 u/sec=23.39	ppl=7.854298 
[2023-12-17:12:01:27:INFO:sockeye.training:__call__] E=14 B=56400	s/sec=2110.53 tok/sec=99284.06 u/sec=23.22	ppl=7.865973 
[2023-12-17:12:01:29:INFO:sockeye.training:__call__] E=14 B=56450	s/sec=2151.85 tok/sec=99200.12 u/sec=23.15	ppl=7.883930 
[2023-12-17:12:01:31:INFO:sockeye.training:__call__] E=14 B=56500	s/sec=2158.45 tok/sec=98229.99 u/sec=23.46	ppl=7.892618 
[2023-12-17:12:01:33:INFO:sockeye.training:__call__] E=14 B=56550	s/sec=2139.53 tok/sec=99226.74 u/sec=23.18	ppl=7.891532 
[2023-12-17:12:01:35:INFO:sockeye.training:__call__] E=14 B=56600	s/sec=2250.39 tok/sec=94914.50 u/sec=23.92	ppl=7.917576 
[2023-12-17:12:01:38:INFO:sockeye.training:__call__] E=14 B=56650	s/sec=2102.95 tok/sec=100345.18 u/sec=23.30	ppl=7.928934 
[2023-12-17:12:01:40:INFO:sockeye.training:__call__] E=14 B=56700	s/sec=1872.58 tok/sec=100805.24 u/sec=22.81	ppl=7.938649 
[2023-12-17:12:01:42:INFO:sockeye.training:__call__] E=14 B=56750	s/sec=2160.80 tok/sec=98443.07 u/sec=23.28	ppl=7.951790 
[2023-12-17:12:01:44:INFO:sockeye.training:__call__] E=14 B=56800	s/sec=2032.27 tok/sec=98804.18 u/sec=23.35	ppl=7.957392 
[2023-12-17:12:01:46:INFO:sockeye.training:__call__] E=14 B=56850	s/sec=2012.54 tok/sec=103226.62 u/sec=22.50	ppl=7.967709 
[2023-12-17:12:01:49:INFO:sockeye.training:__call__] E=14 B=56900	s/sec=1921.06 tok/sec=108906.37 u/sec=21.59	ppl=7.976547 
[2023-12-17:12:01:51:INFO:sockeye.training:__call__] E=14 B=56950	s/sec=2153.03 tok/sec=96811.18 u/sec=23.53	ppl=7.989423 
[2023-12-17:12:01:53:INFO:sockeye.training:__call__] E=14 B=57000	s/sec=2174.22 tok/sec=99557.37 u/sec=23.03	ppl=8.009395 
[2023-12-17:12:01:55:INFO:sockeye.training:__call__] E=14 B=57050	s/sec=2064.98 tok/sec=99416.38 u/sec=23.21	ppl=8.022475 
[2023-12-17:12:01:57:INFO:sockeye.training:__call__] E=14 B=57100	s/sec=2043.62 tok/sec=98688.47 u/sec=23.22	ppl=8.034002 
[2023-12-17:12:01:59:INFO:sockeye.training:__call__] E=14 B=57150	s/sec=2065.73 tok/sec=98127.97 u/sec=23.60	ppl=8.049851 
[2023-12-17:12:02:02:INFO:sockeye.training:__call__] E=14 B=57200	s/sec=1857.81 tok/sec=101977.02 u/sec=22.99	ppl=8.054651 
[2023-12-17:12:02:04:INFO:sockeye.training:__call__] E=14 B=57250	s/sec=2260.79 tok/sec=95681.38 u/sec=23.71	ppl=8.076533 
[2023-12-17:12:02:06:INFO:sockeye.training:__call__] E=14 B=57300	s/sec=2031.27 tok/sec=100090.19 u/sec=23.25	ppl=8.091497 
[2023-12-17:12:02:08:INFO:sockeye.training:__call__] E=14 B=57350	s/sec=1923.44 tok/sec=101733.65 u/sec=22.90	ppl=8.101794 
[2023-12-17:12:02:10:INFO:sockeye.training:__call__] E=14 B=57400	s/sec=2145.48 tok/sec=95273.12 u/sec=23.90	ppl=8.117289 
[2023-12-17:12:02:12:INFO:sockeye.training:__call__] E=14 B=57450	s/sec=1949.84 tok/sec=102024.75 u/sec=22.48	ppl=8.127179 
[2023-12-17:12:02:15:INFO:sockeye.training:__call__] E=14 B=57500	s/sec=1848.09 tok/sec=101355.67 u/sec=22.87	ppl=8.137909 
[2023-12-17:12:02:17:INFO:sockeye.training:__call__] E=14 B=57550	s/sec=2123.92 tok/sec=97528.52 u/sec=23.33	ppl=8.152834 
[2023-12-17:12:02:19:INFO:sockeye.training:__call__] E=14 B=57600	s/sec=2229.60 tok/sec=97177.64 u/sec=23.30	ppl=8.163797 
[2023-12-17:12:02:21:INFO:sockeye.training:__call__] E=14 B=57650	s/sec=2003.54 tok/sec=101195.55 u/sec=23.23	ppl=8.172816 
[2023-12-17:12:02:23:INFO:sockeye.training:__call__] E=14 B=57700	s/sec=2143.76 tok/sec=98456.60 u/sec=23.30	ppl=8.183285 
[2023-12-17:12:02:25:INFO:sockeye.training:__call__] E=14 B=57750	s/sec=2022.45 tok/sec=99316.68 u/sec=23.32	ppl=8.192515 
[2023-12-17:12:02:27:INFO:sockeye.training:__call__] E=14 B=57800	s/sec=1881.87 tok/sec=101236.57 u/sec=22.66	ppl=8.205665 
[2023-12-17:12:02:30:INFO:sockeye.training:__call__] E=14 B=57850	s/sec=2182.22 tok/sec=96731.38 u/sec=23.72	ppl=8.217166 
[2023-12-17:12:02:32:INFO:sockeye.training:__call__] E=14 B=57900	s/sec=2164.68 tok/sec=97019.73 u/sec=23.49	ppl=8.228428 
[2023-12-17:12:02:34:INFO:sockeye.training:__call__] E=14 B=57950	s/sec=2162.92 tok/sec=98959.79 u/sec=22.87	ppl=8.239674 
[2023-12-17:12:02:36:INFO:sockeye.training:__call__] E=14 B=58000	s/sec=2056.13 tok/sec=98664.56 u/sec=23.20	ppl=8.250584 
[2023-12-17:12:02:38:INFO:sockeye.training:__call__] E=14 B=58050	s/sec=1892.03 tok/sec=102337.09 u/sec=22.78	ppl=8.260418 
[2023-12-17:12:02:40:INFO:sockeye.training:__call__] E=14 B=58100	s/sec=2195.10 tok/sec=97663.15 u/sec=23.41	ppl=8.267797 
[2023-12-17:12:02:43:INFO:sockeye.training:__call__] E=14 B=58150	s/sec=1818.45 tok/sec=101513.90 u/sec=22.69	ppl=8.273512 
[2023-12-17:12:02:45:INFO:sockeye.training:__call__] E=14 B=58200	s/sec=1985.97 tok/sec=99738.41 u/sec=22.53	ppl=8.285084 
[2023-12-17:12:02:47:INFO:sockeye.training:__call__] E=14 B=58250	s/sec=1878.56 tok/sec=101565.61 u/sec=22.71	ppl=8.295325 
[2023-12-17:12:02:49:INFO:sockeye.training:__call__] E=14 B=58300	s/sec=2110.14 tok/sec=98220.21 u/sec=22.82	ppl=8.305088 
[2023-12-17:12:02:51:INFO:sockeye.training:__call__] E=14 B=58350	s/sec=2081.85 tok/sec=98061.83 u/sec=23.15	ppl=8.315913 
[2023-12-17:12:02:54:INFO:sockeye.training:__call__] E=14 B=58400	s/sec=2089.45 tok/sec=97851.00 u/sec=22.99	ppl=8.324393 
[2023-12-17:12:02:56:INFO:sockeye.training:__call__] E=14 B=58450	s/sec=1889.41 tok/sec=102435.35 u/sec=22.71	ppl=8.329757 
[2023-12-17:12:02:58:INFO:sockeye.training:__call__] E=14 B=58500	s/sec=2318.17 tok/sec=92726.86 u/sec=23.99	ppl=8.339025 
[2023-12-17:12:03:00:INFO:sockeye.training:__call__] E=14 B=58550	s/sec=2129.03 tok/sec=98834.91 u/sec=23.43	ppl=8.349818 
[2023-12-17:12:03:02:INFO:sockeye.training:__call__] E=14 B=58600	s/sec=2030.58 tok/sec=98955.23 u/sec=23.24	ppl=8.358386 
[2023-12-17:12:03:04:INFO:sockeye.training:__call__] E=14 B=58650	s/sec=2195.92 tok/sec=99191.58 u/sec=23.22	ppl=8.367518 
[2023-12-17:12:03:06:INFO:sockeye.training:__call__] E=14 B=58700	s/sec=1855.65 tok/sec=100402.99 u/sec=22.88	ppl=8.373584 
[2023-12-17:12:03:09:INFO:sockeye.training:__call__] E=14 B=58750	s/sec=2086.95 tok/sec=97741.39 u/sec=23.46	ppl=8.381176 
[2023-12-17:12:03:11:INFO:sockeye.training:__call__] E=14 B=58800	s/sec=2232.72 tok/sec=96647.77 u/sec=23.69	ppl=8.392150 
[2023-12-17:12:03:13:INFO:sockeye.training:__call__] E=14 B=58850	s/sec=1934.00 tok/sec=101118.72 u/sec=22.64	ppl=8.398900 
[2023-12-17:12:03:15:INFO:sockeye.training:__call__] E=14 B=58900	s/sec=2094.78 tok/sec=98525.37 u/sec=23.25	ppl=8.409974 
[2023-12-17:12:03:17:INFO:sockeye.training:__call__] E=14 B=58950	s/sec=2232.90 tok/sec=96128.42 u/sec=23.65	ppl=8.420638 
[2023-12-17:12:03:19:INFO:sockeye.training:__call__] E=14 B=59000	s/sec=1955.29 tok/sec=98454.35 u/sec=23.19	ppl=8.428989 
[2023-12-17:12:03:21:INFO:sockeye.training:__call__] E=14 B=59050	s/sec=2177.92 tok/sec=98811.70 u/sec=23.19	ppl=8.439351 
[2023-12-17:12:03:24:INFO:sockeye.training:__call__] E=14 B=59100	s/sec=1848.64 tok/sec=103872.56 u/sec=22.70	ppl=8.445466 
[2023-12-17:12:03:26:INFO:sockeye.training:__call__] E=14 B=59150	s/sec=1871.83 tok/sec=100781.88 u/sec=23.21	ppl=8.452075 
[2023-12-17:12:03:28:INFO:sockeye.training:__call__] E=14 B=59200	s/sec=2058.88 tok/sec=98679.41 u/sec=22.98	ppl=8.461119 
[2023-12-17:12:03:30:INFO:sockeye.training:__call__] E=14 B=59250	s/sec=2088.48 tok/sec=98270.85 u/sec=23.39	ppl=8.471642 
[2023-12-17:12:03:32:INFO:sockeye.training:__call__] E=14 B=59300	s/sec=2252.88 tok/sec=97007.99 u/sec=23.31	ppl=8.481534 
[2023-12-17:12:03:34:INFO:sockeye.training:__call__] E=14 B=59350	s/sec=2252.54 tok/sec=95346.28 u/sec=23.82	ppl=8.490975 
[2023-12-17:12:03:37:INFO:sockeye.training:__call__] E=14 B=59400	s/sec=1816.89 tok/sec=105838.47 u/sec=22.05	ppl=8.494151 
[2023-12-17:12:03:39:INFO:sockeye.training:__call__] E=14 B=59450	s/sec=1978.21 tok/sec=99648.45 u/sec=23.07	ppl=8.501294 
[2023-12-17:12:03:41:INFO:sockeye.training:__call__] E=14 B=59500	s/sec=2160.91 tok/sec=97555.92 u/sec=23.17	ppl=8.511319 
[2023-12-17:12:03:43:INFO:sockeye.training:__call__] E=14 B=59550	s/sec=1778.45 tok/sec=102515.59 u/sec=23.06	ppl=8.517742 
[2023-12-17:12:03:45:INFO:sockeye.training:__call__] E=14 B=59600	s/sec=2114.11 tok/sec=96212.80 u/sec=23.64	ppl=8.523866 
[2023-12-17:12:03:47:INFO:sockeye.training:__call__] E=14 B=59650	s/sec=2210.49 tok/sec=97367.50 u/sec=23.62	ppl=8.533241 
[2023-12-17:12:03:49:INFO:sockeye.training:__call__] E=14 B=59700	s/sec=2089.60 tok/sec=97143.94 u/sec=23.49	ppl=8.538282 
[2023-12-17:12:03:52:INFO:sockeye.training:__call__] E=14 B=59750	s/sec=2069.52 tok/sec=100694.47 u/sec=23.06	ppl=8.546870 
[2023-12-17:12:03:54:INFO:sockeye.training:__call__] E=14 B=59800	s/sec=2161.06 tok/sec=98367.37 u/sec=23.41	ppl=8.553414 
[2023-12-17:12:03:56:INFO:sockeye.training:__call__] E=14 B=59850	s/sec=2124.19 tok/sec=100056.23 u/sec=23.25	ppl=8.559715 
[2023-12-17:12:03:58:INFO:sockeye.training:__call__] E=14 B=59900	s/sec=2098.85 tok/sec=97986.49 u/sec=23.42	ppl=8.566890 
[2023-12-17:12:04:00:INFO:sockeye.training:__call__] E=15 B=59950	s/sec=2047.44 tok/sec=92059.16 u/sec=22.49	ppl=8.562251 
[2023-12-17:12:04:02:INFO:sockeye.training:__call__] E=15 B=60000	s/sec=1954.92 tok/sec=101847.52 u/sec=23.01	ppl=8.545496 
[2023-12-17:12:04:02:INFO:sockeye.training:_create_checkpoint] Checkpoint [15]	Updates=60000 Epoch=15 Samples=5346912 Time-cost=172.743 Updates/sec=23.156
[2023-12-17:12:04:02:INFO:sockeye.training:_create_checkpoint] Checkpoint [15]	Train-ppl=8.545496
[2023-12-17:12:04:05:INFO:sockeye.training:_evaluate] Checkpoint [15]	Validation-ppl=28.687923	Validation-bleu=0.175246	Validation-chrf=31.592098	Validation-rouge1=0.170929	Validation-rouge2=0.082893	Validation-rougel=0.164011	Validation-length-ratio-mse=1.281912	Validation-ter=93.809327	Validation-avg-sec-per-sent=0.003454	Validation-decode-walltime=1.726968
[2023-12-17:12:04:05:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 10 checkpoints, best so far: 31.983236
[2023-12-17:12:04:05:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [15]	Learning-rate=0.000180
[2023-12-17:12:04:06:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:04:06:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:04:06:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00015"
[2023-12-17:12:04:08:INFO:sockeye.training:__call__] E=15 B=60050	s/sec=739.91 tok/sec=39952.21 u/sec=9.05	ppl=7.241315 
[2023-12-17:12:04:10:INFO:sockeye.training:__call__] E=15 B=60100	s/sec=2211.20 tok/sec=98594.69 u/sec=23.38	ppl=7.337469 
[2023-12-17:12:04:12:INFO:sockeye.training:__call__] E=15 B=60150	s/sec=2095.55 tok/sec=97949.20 u/sec=23.68	ppl=7.351622 
[2023-12-17:12:04:14:INFO:sockeye.training:__call__] E=15 B=60200	s/sec=1861.98 tok/sec=101904.99 u/sec=22.82	ppl=7.335726 
[2023-12-17:12:04:17:INFO:sockeye.training:__call__] E=15 B=60250	s/sec=2316.89 tok/sec=94568.67 u/sec=23.86	ppl=7.383881 
[2023-12-17:12:04:19:INFO:sockeye.training:__call__] E=15 B=60300	s/sec=2017.57 tok/sec=97537.13 u/sec=23.57	ppl=7.399723 
[2023-12-17:12:04:21:INFO:sockeye.training:__call__] E=15 B=60350	s/sec=1993.67 tok/sec=100099.05 u/sec=22.78	ppl=7.434946 
[2023-12-17:12:04:23:INFO:sockeye.training:__call__] E=15 B=60400	s/sec=1943.45 tok/sec=99982.21 u/sec=22.75	ppl=7.450035 
[2023-12-17:12:04:25:INFO:sockeye.training:__call__] E=15 B=60450	s/sec=1922.83 tok/sec=100984.09 u/sec=23.25	ppl=7.462857 
[2023-12-17:12:04:27:INFO:sockeye.training:__call__] E=15 B=60500	s/sec=2025.89 tok/sec=102769.77 u/sec=22.49	ppl=7.475374 
[2023-12-17:12:04:30:INFO:sockeye.training:__call__] E=15 B=60550	s/sec=1910.39 tok/sec=98474.66 u/sec=23.32	ppl=7.482907 
[2023-12-17:12:04:32:INFO:sockeye.training:__call__] E=15 B=60600	s/sec=2267.29 tok/sec=96538.66 u/sec=23.31	ppl=7.513902 
[2023-12-17:12:04:34:INFO:sockeye.training:__call__] E=15 B=60650	s/sec=2013.94 tok/sec=102225.07 u/sec=22.84	ppl=7.525573 
[2023-12-17:12:04:36:INFO:sockeye.training:__call__] E=15 B=60700	s/sec=2213.90 tok/sec=95905.43 u/sec=23.53	ppl=7.545948 
[2023-12-17:12:04:38:INFO:sockeye.training:__call__] E=15 B=60750	s/sec=2033.39 tok/sec=100867.70 u/sec=22.98	ppl=7.552270 
[2023-12-17:12:04:40:INFO:sockeye.training:__call__] E=15 B=60800	s/sec=2060.76 tok/sec=100490.70 u/sec=23.21	ppl=7.569525 
[2023-12-17:12:04:43:INFO:sockeye.training:__call__] E=15 B=60850	s/sec=1798.87 tok/sec=105036.56 u/sec=22.40	ppl=7.571250 
[2023-12-17:12:04:45:INFO:sockeye.training:__call__] E=15 B=60900	s/sec=2187.80 tok/sec=95991.65 u/sec=23.58	ppl=7.589757 
[2023-12-17:12:04:47:INFO:sockeye.training:__call__] E=15 B=60950	s/sec=2003.88 tok/sec=100075.38 u/sec=23.19	ppl=7.601091 
[2023-12-17:12:04:49:INFO:sockeye.training:__call__] E=15 B=61000	s/sec=2055.28 tok/sec=98473.28 u/sec=23.48	ppl=7.607634 
[2023-12-17:12:04:51:INFO:sockeye.training:__call__] E=15 B=61050	s/sec=1932.61 tok/sec=101182.77 u/sec=22.83	ppl=7.622161 
[2023-12-17:12:04:53:INFO:sockeye.training:__call__] E=15 B=61100	s/sec=2373.24 tok/sec=96318.25 u/sec=23.58	ppl=7.644777 
[2023-12-17:12:04:55:INFO:sockeye.training:__call__] E=15 B=61150	s/sec=2151.73 tok/sec=97456.49 u/sec=23.11	ppl=7.657986 
[2023-12-17:12:04:58:INFO:sockeye.training:__call__] E=15 B=61200	s/sec=2176.46 tok/sec=97535.27 u/sec=23.25	ppl=7.677365 
[2023-12-17:12:05:00:INFO:sockeye.training:__call__] E=15 B=61250	s/sec=2110.29 tok/sec=98030.08 u/sec=23.18	ppl=7.697934 
[2023-12-17:12:05:02:INFO:sockeye.training:__call__] E=15 B=61300	s/sec=2016.91 tok/sec=100103.99 u/sec=23.17	ppl=7.707455 
[2023-12-17:12:05:04:INFO:sockeye.training:__call__] E=15 B=61350	s/sec=2201.18 tok/sec=97969.24 u/sec=23.28	ppl=7.722849 
[2023-12-17:12:05:06:INFO:sockeye.training:__call__] E=15 B=61400	s/sec=1941.49 tok/sec=100043.17 u/sec=22.68	ppl=7.735101 
[2023-12-17:12:05:08:INFO:sockeye.training:__call__] E=15 B=61450	s/sec=2208.09 tok/sec=96613.39 u/sec=23.55	ppl=7.750912 
[2023-12-17:12:05:11:INFO:sockeye.training:__call__] E=15 B=61500	s/sec=2062.99 tok/sec=99780.08 u/sec=22.74	ppl=7.760770 
[2023-12-17:12:05:13:INFO:sockeye.training:__call__] E=15 B=61550	s/sec=1948.03 tok/sec=98403.34 u/sec=23.19	ppl=7.771724 
[2023-12-17:12:05:15:INFO:sockeye.training:__call__] E=15 B=61600	s/sec=2175.09 tok/sec=100172.37 u/sec=23.12	ppl=7.784100 
[2023-12-17:12:05:17:INFO:sockeye.training:__call__] E=15 B=61650	s/sec=2106.67 tok/sec=97568.95 u/sec=23.51	ppl=7.799408 
[2023-12-17:12:05:19:INFO:sockeye.training:__call__] E=15 B=61700	s/sec=1954.36 tok/sec=97970.75 u/sec=23.22	ppl=7.806339 
[2023-12-17:12:05:21:INFO:sockeye.training:__call__] E=15 B=61750	s/sec=2190.77 tok/sec=97506.16 u/sec=23.17	ppl=7.821372 
[2023-12-17:12:05:23:INFO:sockeye.training:__call__] E=15 B=61800	s/sec=2256.71 tok/sec=94690.34 u/sec=23.83	ppl=7.836796 
[2023-12-17:12:05:26:INFO:sockeye.training:__call__] E=15 B=61850	s/sec=2121.82 tok/sec=99067.59 u/sec=23.10	ppl=7.843817 
[2023-12-17:12:05:28:INFO:sockeye.training:__call__] E=15 B=61900	s/sec=2187.41 tok/sec=96744.60 u/sec=23.61	ppl=7.855686 
[2023-12-17:12:05:30:INFO:sockeye.training:__call__] E=15 B=61950	s/sec=2031.72 tok/sec=101085.25 u/sec=23.00	ppl=7.863180 
[2023-12-17:12:05:32:INFO:sockeye.training:__call__] E=15 B=62000	s/sec=2080.50 tok/sec=99036.17 u/sec=23.10	ppl=7.871572 
[2023-12-17:12:05:34:INFO:sockeye.training:__call__] E=15 B=62050	s/sec=1947.81 tok/sec=101963.05 u/sec=22.50	ppl=7.880499 
[2023-12-17:12:05:36:INFO:sockeye.training:__call__] E=15 B=62100	s/sec=2196.96 tok/sec=98625.54 u/sec=23.19	ppl=7.890215 
[2023-12-17:12:05:39:INFO:sockeye.training:__call__] E=15 B=62150	s/sec=2120.47 tok/sec=99804.25 u/sec=23.42	ppl=7.898319 
[2023-12-17:12:05:41:INFO:sockeye.training:__call__] E=15 B=62200	s/sec=2140.23 tok/sec=99692.51 u/sec=22.83	ppl=7.909344 
[2023-12-17:12:05:43:INFO:sockeye.training:__call__] E=15 B=62250	s/sec=1988.10 tok/sec=98627.54 u/sec=23.14	ppl=7.919041 
[2023-12-17:12:05:45:INFO:sockeye.training:__call__] E=15 B=62300	s/sec=1930.16 tok/sec=98235.30 u/sec=23.47	ppl=7.924142 
[2023-12-17:12:05:47:INFO:sockeye.training:__call__] E=15 B=62350	s/sec=1910.27 tok/sec=99932.90 u/sec=22.83	ppl=7.929715 
[2023-12-17:12:05:49:INFO:sockeye.training:__call__] E=15 B=62400	s/sec=1962.95 tok/sec=98498.78 u/sec=23.37	ppl=7.935829 
[2023-12-17:12:05:52:INFO:sockeye.training:__call__] E=15 B=62450	s/sec=2063.08 tok/sec=94671.74 u/sec=23.97	ppl=7.946471 
[2023-12-17:12:05:54:INFO:sockeye.training:__call__] E=15 B=62500	s/sec=1972.31 tok/sec=98882.57 u/sec=23.17	ppl=7.952629 
[2023-12-17:12:05:56:INFO:sockeye.training:__call__] E=15 B=62550	s/sec=1936.72 tok/sec=98963.26 u/sec=23.32	ppl=7.957210 
[2023-12-17:12:05:58:INFO:sockeye.training:__call__] E=15 B=62600	s/sec=1917.50 tok/sec=102680.33 u/sec=23.09	ppl=7.960891 
[2023-12-17:12:06:00:INFO:sockeye.training:__call__] E=15 B=62650	s/sec=2066.88 tok/sec=100127.18 u/sec=23.11	ppl=7.970909 
[2023-12-17:12:06:02:INFO:sockeye.training:__call__] E=15 B=62700	s/sec=2255.04 tok/sec=97087.15 u/sec=23.49	ppl=7.979869 
[2023-12-17:12:06:04:INFO:sockeye.training:__call__] E=15 B=62750	s/sec=1903.69 tok/sec=98722.05 u/sec=23.42	ppl=7.987676 
[2023-12-17:12:06:07:INFO:sockeye.training:__call__] E=15 B=62800	s/sec=2213.37 tok/sec=95809.80 u/sec=23.49	ppl=7.998996 
[2023-12-17:12:06:09:INFO:sockeye.training:__call__] E=15 B=62850	s/sec=2159.81 tok/sec=97961.27 u/sec=23.48	ppl=8.008504 
[2023-12-17:12:06:11:INFO:sockeye.training:__call__] E=15 B=62900	s/sec=1858.62 tok/sec=102012.79 u/sec=22.78	ppl=8.016146 
[2023-12-17:12:06:13:INFO:sockeye.training:__call__] E=15 B=62950	s/sec=2008.76 tok/sec=100066.18 u/sec=22.79	ppl=8.023006 
[2023-12-17:12:06:15:INFO:sockeye.training:__call__] E=15 B=63000	s/sec=2238.60 tok/sec=97352.66 u/sec=23.55	ppl=8.030843 
[2023-12-17:12:06:17:INFO:sockeye.training:__call__] E=15 B=63050	s/sec=1827.96 tok/sec=102629.41 u/sec=22.90	ppl=8.034061 
[2023-12-17:12:06:19:INFO:sockeye.training:__call__] E=15 B=63100	s/sec=2378.27 tok/sec=94949.76 u/sec=23.59	ppl=8.044457 
[2023-12-17:12:06:22:INFO:sockeye.training:__call__] E=15 B=63150	s/sec=1874.01 tok/sec=103913.62 u/sec=21.77	ppl=8.051647 
[2023-12-17:12:06:24:INFO:sockeye.training:__call__] E=15 B=63200	s/sec=1887.70 tok/sec=107592.14 u/sec=21.30	ppl=8.057205 
[2023-12-17:12:06:26:INFO:sockeye.training:__call__] E=15 B=63250	s/sec=2110.75 tok/sec=99420.21 u/sec=23.14	ppl=8.066729 
[2023-12-17:12:06:28:INFO:sockeye.training:__call__] E=15 B=63300	s/sec=2084.32 tok/sec=99782.92 u/sec=22.98	ppl=8.075161 
[2023-12-17:12:06:31:INFO:sockeye.training:__call__] E=15 B=63350	s/sec=2123.87 tok/sec=98500.00 u/sec=23.21	ppl=8.083577 
[2023-12-17:12:06:33:INFO:sockeye.training:__call__] E=15 B=63400	s/sec=1995.24 tok/sec=99858.80 u/sec=23.31	ppl=8.091953 
[2023-12-17:12:06:35:INFO:sockeye.training:__call__] E=15 B=63450	s/sec=2181.58 tok/sec=98345.33 u/sec=23.15	ppl=8.101842 
[2023-12-17:12:06:37:INFO:sockeye.training:__call__] E=15 B=63500	s/sec=2209.42 tok/sec=97244.43 u/sec=23.40	ppl=8.111488 
[2023-12-17:12:06:39:INFO:sockeye.training:__call__] E=15 B=63550	s/sec=2074.31 tok/sec=98201.32 u/sec=23.19	ppl=8.119740 
[2023-12-17:12:06:41:INFO:sockeye.training:__call__] E=15 B=63600	s/sec=2310.29 tok/sec=94036.13 u/sec=23.95	ppl=8.129783 
[2023-12-17:12:06:43:INFO:sockeye.training:__call__] E=15 B=63650	s/sec=2005.50 tok/sec=99684.99 u/sec=23.04	ppl=8.136766 
[2023-12-17:12:06:46:INFO:sockeye.training:__call__] E=15 B=63700	s/sec=2162.38 tok/sec=100780.48 u/sec=22.64	ppl=8.144803 
[2023-12-17:12:06:48:INFO:sockeye.training:__call__] E=15 B=63750	s/sec=1976.03 tok/sec=100187.33 u/sec=22.91	ppl=8.150624 
[2023-12-17:12:06:50:INFO:sockeye.training:__call__] E=15 B=63800	s/sec=2152.50 tok/sec=94572.30 u/sec=23.90	ppl=8.158553 
[2023-12-17:12:06:52:INFO:sockeye.training:__call__] E=15 B=63850	s/sec=1992.86 tok/sec=101876.95 u/sec=22.81	ppl=8.164673 
[2023-12-17:12:06:54:INFO:sockeye.training:__call__] E=15 B=63900	s/sec=2166.79 tok/sec=94927.60 u/sec=23.80	ppl=8.170475 
[2023-12-17:12:06:57:INFO:sockeye.training:__call__] E=16 B=63950	s/sec=1829.39 tok/sec=97426.46 u/sec=22.03	ppl=8.162186 
[2023-12-17:12:06:59:INFO:sockeye.training:__call__] E=16 B=64000	s/sec=2014.72 tok/sec=99556.10 u/sec=23.19	ppl=8.147151 
[2023-12-17:12:06:59:INFO:sockeye.training:_create_checkpoint] Checkpoint [16]	Updates=64000 Epoch=16 Samples=5703512 Time-cost=172.853 Updates/sec=23.141
[2023-12-17:12:06:59:INFO:sockeye.training:_create_checkpoint] Checkpoint [16]	Train-ppl=8.147151
[2023-12-17:12:07:01:INFO:sockeye.training:_evaluate] Checkpoint [16]	Validation-ppl=29.548111	Validation-bleu=0.171251	Validation-chrf=31.124228	Validation-rouge1=0.153663	Validation-rouge2=0.076671	Validation-rougel=0.147653	Validation-length-ratio-mse=1.300916	Validation-ter=96.987206	Validation-avg-sec-per-sent=0.003525	Validation-decode-walltime=1.762334
[2023-12-17:12:07:01:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 11 checkpoints, best so far: 31.983236
[2023-12-17:12:07:01:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [16]	Learning-rate=0.000180
[2023-12-17:12:07:02:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:07:02:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:07:02:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00016"
[2023-12-17:12:07:04:INFO:sockeye.training:__call__] E=16 B=64050	s/sec=767.37 tok/sec=39434.83 u/sec=9.03	ppl=7.019364 
[2023-12-17:12:07:06:INFO:sockeye.training:__call__] E=16 B=64100	s/sec=1952.42 tok/sec=101891.90 u/sec=22.89	ppl=7.034194 
[2023-12-17:12:07:09:INFO:sockeye.training:__call__] E=16 B=64150	s/sec=2124.02 tok/sec=98417.97 u/sec=23.21	ppl=7.083818 
[2023-12-17:12:07:11:INFO:sockeye.training:__call__] E=16 B=64200	s/sec=1941.06 tok/sec=99658.34 u/sec=22.93	ppl=7.075590 
[2023-12-17:12:07:13:INFO:sockeye.training:__call__] E=16 B=64250	s/sec=1964.15 tok/sec=98229.73 u/sec=23.38	ppl=7.087249 
[2023-12-17:12:07:15:INFO:sockeye.training:__call__] E=16 B=64300	s/sec=1828.93 tok/sec=102655.35 u/sec=22.95	ppl=7.092815 
[2023-12-17:12:07:17:INFO:sockeye.training:__call__] E=16 B=64350	s/sec=1970.41 tok/sec=101463.51 u/sec=22.60	ppl=7.107643 
[2023-12-17:12:07:19:INFO:sockeye.training:__call__] E=16 B=64400	s/sec=2098.94 tok/sec=97885.83 u/sec=23.30	ppl=7.118538 
[2023-12-17:12:07:22:INFO:sockeye.training:__call__] E=16 B=64450	s/sec=1893.16 tok/sec=102369.24 u/sec=22.80	ppl=7.133476 
[2023-12-17:12:07:24:INFO:sockeye.training:__call__] E=16 B=64500	s/sec=2035.96 tok/sec=100669.13 u/sec=22.76	ppl=7.156145 
[2023-12-17:12:07:26:INFO:sockeye.training:__call__] E=16 B=64550	s/sec=2158.97 tok/sec=99628.78 u/sec=22.99	ppl=7.182879 
[2023-12-17:12:07:28:INFO:sockeye.training:__call__] E=16 B=64600	s/sec=2126.89 tok/sec=98224.85 u/sec=22.88	ppl=7.199760 
[2023-12-17:12:07:30:INFO:sockeye.training:__call__] E=16 B=64650	s/sec=2343.78 tok/sec=96124.92 u/sec=23.48	ppl=7.221750 
[2023-12-17:12:07:32:INFO:sockeye.training:__call__] E=16 B=64700	s/sec=1960.82 tok/sec=100846.49 u/sec=22.95	ppl=7.229830 
[2023-12-17:12:07:35:INFO:sockeye.training:__call__] E=16 B=64750	s/sec=1980.93 tok/sec=98373.83 u/sec=23.36	ppl=7.236368 
[2023-12-17:12:07:37:INFO:sockeye.training:__call__] E=16 B=64800	s/sec=1992.43 tok/sec=99503.37 u/sec=23.06	ppl=7.242851 
[2023-12-17:12:07:39:INFO:sockeye.training:__call__] E=16 B=64850	s/sec=1910.93 tok/sec=104190.66 u/sec=22.49	ppl=7.250413 
[2023-12-17:12:07:41:INFO:sockeye.training:__call__] E=16 B=64900	s/sec=2227.77 tok/sec=95820.79 u/sec=23.72	ppl=7.273376 
[2023-12-17:12:07:43:INFO:sockeye.training:__call__] E=16 B=64950	s/sec=2146.66 tok/sec=99605.04 u/sec=22.93	ppl=7.291188 
[2023-12-17:12:07:45:INFO:sockeye.training:__call__] E=16 B=65000	s/sec=2094.55 tok/sec=99290.82 u/sec=23.21	ppl=7.306971 
[2023-12-17:12:07:48:INFO:sockeye.training:__call__] E=16 B=65050	s/sec=2031.23 tok/sec=95883.19 u/sec=23.38	ppl=7.319520 
[2023-12-17:12:07:50:INFO:sockeye.training:__call__] E=16 B=65100	s/sec=2070.91 tok/sec=97407.35 u/sec=23.28	ppl=7.328948 
[2023-12-17:12:07:52:INFO:sockeye.training:__call__] E=16 B=65150	s/sec=1988.79 tok/sec=100952.36 u/sec=23.06	ppl=7.340756 
[2023-12-17:12:07:54:INFO:sockeye.training:__call__] E=16 B=65200	s/sec=1888.41 tok/sec=99613.66 u/sec=23.05	ppl=7.358647 
[2023-12-17:12:07:56:INFO:sockeye.training:__call__] E=16 B=65250	s/sec=1834.63 tok/sec=98237.04 u/sec=23.45	ppl=7.359930 
[2023-12-17:12:07:58:INFO:sockeye.training:__call__] E=16 B=65300	s/sec=2168.56 tok/sec=99470.56 u/sec=23.29	ppl=7.373586 
[2023-12-17:12:08:01:INFO:sockeye.training:__call__] E=16 B=65350	s/sec=1831.21 tok/sec=102978.45 u/sec=22.44	ppl=7.379293 
[2023-12-17:12:08:03:INFO:sockeye.training:__call__] E=16 B=65400	s/sec=2034.72 tok/sec=98877.59 u/sec=23.08	ppl=7.393728 
[2023-12-17:12:08:05:INFO:sockeye.training:__call__] E=16 B=65450	s/sec=2076.39 tok/sec=99637.23 u/sec=23.13	ppl=7.404337 
[2023-12-17:12:08:07:INFO:sockeye.training:__call__] E=16 B=65500	s/sec=2433.08 tok/sec=93975.62 u/sec=24.21	ppl=7.422916 
[2023-12-17:12:08:09:INFO:sockeye.training:__call__] E=16 B=65550	s/sec=2170.99 tok/sec=97675.53 u/sec=23.52	ppl=7.434070 
[2023-12-17:12:08:11:INFO:sockeye.training:__call__] E=16 B=65600	s/sec=2091.14 tok/sec=105839.78 u/sec=21.78	ppl=7.442864 
[2023-12-17:12:08:13:INFO:sockeye.training:__call__] E=16 B=65650	s/sec=2244.23 tok/sec=93966.51 u/sec=23.94	ppl=7.456539 
[2023-12-17:12:08:16:INFO:sockeye.training:__call__] E=16 B=65700	s/sec=2077.76 tok/sec=100408.08 u/sec=22.94	ppl=7.470398 
[2023-12-17:12:08:18:INFO:sockeye.training:__call__] E=16 B=65750	s/sec=2048.66 tok/sec=97260.75 u/sec=23.32	ppl=7.477808 
[2023-12-17:12:08:20:INFO:sockeye.training:__call__] E=16 B=65800	s/sec=2180.72 tok/sec=97378.04 u/sec=23.46	ppl=7.488259 
[2023-12-17:12:08:22:INFO:sockeye.training:__call__] E=16 B=65850	s/sec=2098.96 tok/sec=98610.41 u/sec=23.22	ppl=7.497932 
[2023-12-17:12:08:24:INFO:sockeye.training:__call__] E=16 B=65900	s/sec=2420.22 tok/sec=92264.13 u/sec=24.32	ppl=7.512657 
[2023-12-17:12:08:26:INFO:sockeye.training:__call__] E=16 B=65950	s/sec=1988.30 tok/sec=101182.59 u/sec=23.01	ppl=7.521607 
[2023-12-17:12:08:28:INFO:sockeye.training:__call__] E=16 B=66000	s/sec=2289.52 tok/sec=95483.85 u/sec=24.01	ppl=7.537213 
[2023-12-17:12:08:31:INFO:sockeye.training:__call__] E=16 B=66050	s/sec=2260.78 tok/sec=95693.79 u/sec=23.63	ppl=7.547281 
[2023-12-17:12:08:33:INFO:sockeye.training:__call__] E=16 B=66100	s/sec=2331.45 tok/sec=96025.78 u/sec=23.50	ppl=7.560809 
[2023-12-17:12:08:35:INFO:sockeye.training:__call__] E=16 B=66150	s/sec=2041.29 tok/sec=102202.53 u/sec=22.74	ppl=7.568015 
[2023-12-17:12:08:37:INFO:sockeye.training:__call__] E=16 B=66200	s/sec=2183.75 tok/sec=97920.98 u/sec=23.13	ppl=7.579785 
[2023-12-17:12:08:39:INFO:sockeye.training:__call__] E=16 B=66250	s/sec=2350.75 tok/sec=95090.11 u/sec=23.66	ppl=7.590375 
[2023-12-17:12:08:41:INFO:sockeye.training:__call__] E=16 B=66300	s/sec=2087.72 tok/sec=98460.30 u/sec=23.18	ppl=7.599432 
[2023-12-17:12:08:43:INFO:sockeye.training:__call__] E=16 B=66350	s/sec=1923.19 tok/sec=101037.60 u/sec=23.03	ppl=7.607702 
[2023-12-17:12:08:46:INFO:sockeye.training:__call__] E=16 B=66400	s/sec=1933.58 tok/sec=100257.69 u/sec=22.55	ppl=7.617915 
[2023-12-17:12:08:48:INFO:sockeye.training:__call__] E=16 B=66450	s/sec=2157.03 tok/sec=96347.52 u/sec=23.41	ppl=7.624782 
[2023-12-17:12:08:50:INFO:sockeye.training:__call__] E=16 B=66500	s/sec=2138.47 tok/sec=95681.16 u/sec=23.87	ppl=7.634018 
[2023-12-17:12:08:52:INFO:sockeye.training:__call__] E=16 B=66550	s/sec=2162.90 tok/sec=98334.22 u/sec=23.67	ppl=7.644498 
[2023-12-17:12:08:54:INFO:sockeye.training:__call__] E=16 B=66600	s/sec=2013.62 tok/sec=99383.50 u/sec=23.44	ppl=7.654546 
[2023-12-17:12:08:56:INFO:sockeye.training:__call__] E=16 B=66650	s/sec=1927.04 tok/sec=101252.67 u/sec=23.03	ppl=7.659589 
[2023-12-17:12:08:59:INFO:sockeye.training:__call__] E=16 B=66700	s/sec=1945.79 tok/sec=100539.60 u/sec=22.77	ppl=7.664010 
[2023-12-17:12:09:01:INFO:sockeye.training:__call__] E=16 B=66750	s/sec=2163.07 tok/sec=97361.12 u/sec=23.72	ppl=7.670545 
[2023-12-17:12:09:03:INFO:sockeye.training:__call__] E=16 B=66800	s/sec=1997.24 tok/sec=101617.16 u/sec=22.57	ppl=7.679178 
[2023-12-17:12:09:05:INFO:sockeye.training:__call__] E=16 B=66850	s/sec=1904.93 tok/sec=97903.91 u/sec=23.39	ppl=7.685310 
[2023-12-17:12:09:07:INFO:sockeye.training:__call__] E=16 B=66900	s/sec=2016.37 tok/sec=99667.17 u/sec=23.21	ppl=7.693967 
[2023-12-17:12:09:09:INFO:sockeye.training:__call__] E=16 B=66950	s/sec=2106.58 tok/sec=97096.19 u/sec=23.26	ppl=7.702932 
[2023-12-17:12:09:11:INFO:sockeye.training:__call__] E=16 B=67000	s/sec=2176.62 tok/sec=98265.43 u/sec=23.33	ppl=7.710392 
[2023-12-17:12:09:14:INFO:sockeye.training:__call__] E=16 B=67050	s/sec=2189.41 tok/sec=94960.89 u/sec=23.84	ppl=7.717828 
[2023-12-17:12:09:16:INFO:sockeye.training:__call__] E=16 B=67100	s/sec=2037.05 tok/sec=98109.56 u/sec=23.53	ppl=7.724927 
[2023-12-17:12:09:18:INFO:sockeye.training:__call__] E=16 B=67150	s/sec=1941.73 tok/sec=99638.90 u/sec=22.85	ppl=7.731004 
[2023-12-17:12:09:20:INFO:sockeye.training:__call__] E=16 B=67200	s/sec=1973.21 tok/sec=101634.82 u/sec=23.01	ppl=7.738429 
[2023-12-17:12:09:22:INFO:sockeye.training:__call__] E=16 B=67250	s/sec=2055.54 tok/sec=103273.77 u/sec=22.50	ppl=7.744302 
[2023-12-17:12:09:24:INFO:sockeye.training:__call__] E=16 B=67300	s/sec=2227.05 tok/sec=96874.63 u/sec=23.87	ppl=7.753189 
[2023-12-17:12:09:27:INFO:sockeye.training:__call__] E=16 B=67350	s/sec=2091.38 tok/sec=100708.59 u/sec=22.89	ppl=7.761463 
[2023-12-17:12:09:29:INFO:sockeye.training:__call__] E=16 B=67400	s/sec=2107.64 tok/sec=98361.42 u/sec=23.31	ppl=7.769348 
[2023-12-17:12:09:31:INFO:sockeye.training:__call__] E=16 B=67450	s/sec=2022.99 tok/sec=100334.15 u/sec=23.16	ppl=7.774857 
[2023-12-17:12:09:33:INFO:sockeye.training:__call__] E=16 B=67500	s/sec=2127.52 tok/sec=99150.54 u/sec=23.21	ppl=7.782552 
[2023-12-17:12:09:35:INFO:sockeye.training:__call__] E=16 B=67550	s/sec=1981.86 tok/sec=99520.49 u/sec=23.02	ppl=7.791894 
[2023-12-17:12:09:37:INFO:sockeye.training:__call__] E=16 B=67600	s/sec=1953.80 tok/sec=101162.05 u/sec=22.70	ppl=7.798607 
[2023-12-17:12:09:40:INFO:sockeye.training:__call__] E=16 B=67650	s/sec=2071.23 tok/sec=100323.28 u/sec=22.79	ppl=7.805892 
[2023-12-17:12:09:42:INFO:sockeye.training:__call__] E=16 B=67700	s/sec=2130.86 tok/sec=98162.37 u/sec=23.49	ppl=7.813400 
[2023-12-17:12:09:44:INFO:sockeye.training:__call__] E=16 B=67750	s/sec=1861.97 tok/sec=102411.76 u/sec=22.68	ppl=7.818187 
[2023-12-17:12:09:46:INFO:sockeye.training:__call__] E=16 B=67800	s/sec=2110.02 tok/sec=97399.63 u/sec=23.51	ppl=7.823147 
[2023-12-17:12:09:48:INFO:sockeye.training:__call__] E=16 B=67850	s/sec=2054.89 tok/sec=98930.81 u/sec=23.14	ppl=7.829152 
[2023-12-17:12:09:50:INFO:sockeye.training:__call__] E=16 B=67900	s/sec=1957.99 tok/sec=102386.33 u/sec=22.87	ppl=7.834840 
[2023-12-17:12:09:53:INFO:sockeye.training:__call__] E=17 B=67950	s/sec=1751.96 tok/sec=99296.59 u/sec=22.08	ppl=7.827147 
[2023-12-17:12:09:55:INFO:sockeye.training:__call__] E=17 B=68000	s/sec=2085.90 tok/sec=99441.65 u/sec=23.16	ppl=7.814129 
[2023-12-17:12:09:55:INFO:sockeye.training:_create_checkpoint] Checkpoint [17]	Updates=68000 Epoch=17 Samples=6059920 Time-cost=172.734 Updates/sec=23.157
[2023-12-17:12:09:55:INFO:sockeye.training:_create_checkpoint] Checkpoint [17]	Train-ppl=7.814129
[2023-12-17:12:09:57:INFO:sockeye.training:_evaluate] Checkpoint [17]	Validation-ppl=30.078653	Validation-bleu=0.157769	Validation-chrf=30.090398	Validation-rouge1=0.160824	Validation-rouge2=0.079840	Validation-rougel=0.154389	Validation-length-ratio-mse=1.305848	Validation-ter=97.812629	Validation-avg-sec-per-sent=0.003291	Validation-decode-walltime=1.645558
[2023-12-17:12:09:57:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 12 checkpoints, best so far: 31.983236
[2023-12-17:12:09:57:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [17]	Learning-rate=0.000180
[2023-12-17:12:09:58:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:09:58:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:09:58:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00017"
[2023-12-17:12:10:00:INFO:sockeye.training:__call__] E=17 B=68050	s/sec=752.20 tok/sec=39812.21 u/sec=9.16	ppl=6.731433 
[2023-12-17:12:10:02:INFO:sockeye.training:__call__] E=17 B=68100	s/sec=2284.12 tok/sec=96023.98 u/sec=23.64	ppl=6.839886 
[2023-12-17:12:10:04:INFO:sockeye.training:__call__] E=17 B=68150	s/sec=2154.99 tok/sec=99511.80 u/sec=23.42	ppl=6.870842 
[2023-12-17:12:10:07:INFO:sockeye.training:__call__] E=17 B=68200	s/sec=2113.20 tok/sec=100811.93 u/sec=23.13	ppl=6.861070 
[2023-12-17:12:10:09:INFO:sockeye.training:__call__] E=17 B=68250	s/sec=1926.30 tok/sec=101343.74 u/sec=22.98	ppl=6.857021 
[2023-12-17:12:10:11:INFO:sockeye.training:__call__] E=17 B=68300	s/sec=2265.85 tok/sec=95769.59 u/sec=23.88	ppl=6.877688 
[2023-12-17:12:10:13:INFO:sockeye.training:__call__] E=17 B=68350	s/sec=1922.51 tok/sec=100851.47 u/sec=23.33	ppl=6.885332 
[2023-12-17:12:10:15:INFO:sockeye.training:__call__] E=17 B=68400	s/sec=1946.54 tok/sec=102974.76 u/sec=22.66	ppl=6.897742 
[2023-12-17:12:10:17:INFO:sockeye.training:__call__] E=17 B=68450	s/sec=2331.34 tok/sec=96483.88 u/sec=23.81	ppl=6.920167 
[2023-12-17:12:10:20:INFO:sockeye.training:__call__] E=17 B=68500	s/sec=2091.63 tok/sec=97817.24 u/sec=23.18	ppl=6.942617 
[2023-12-17:12:10:22:INFO:sockeye.training:__call__] E=17 B=68550	s/sec=1761.92 tok/sec=104827.37 u/sec=22.38	ppl=6.940193 
[2023-12-17:12:10:24:INFO:sockeye.training:__call__] E=17 B=68600	s/sec=1756.18 tok/sec=102812.33 u/sec=23.11	ppl=6.942563 
[2023-12-17:12:10:26:INFO:sockeye.training:__call__] E=17 B=68650	s/sec=2229.39 tok/sec=98077.76 u/sec=23.82	ppl=6.964135 
[2023-12-17:12:10:28:INFO:sockeye.training:__call__] E=17 B=68700	s/sec=2197.34 tok/sec=95182.42 u/sec=23.93	ppl=6.982443 
[2023-12-17:12:10:30:INFO:sockeye.training:__call__] E=17 B=68750	s/sec=2329.43 tok/sec=95896.11 u/sec=23.87	ppl=7.001259 
[2023-12-17:12:10:32:INFO:sockeye.training:__call__] E=17 B=68800	s/sec=1879.83 tok/sec=103138.49 u/sec=22.51	ppl=7.003365 
[2023-12-17:12:10:35:INFO:sockeye.training:__call__] E=17 B=68850	s/sec=1964.86 tok/sec=98926.63 u/sec=22.25	ppl=7.014911 
[2023-12-17:12:10:37:INFO:sockeye.training:__call__] E=17 B=68900	s/sec=2273.22 tok/sec=97336.71 u/sec=23.84	ppl=7.036502 
[2023-12-17:12:10:39:INFO:sockeye.training:__call__] E=17 B=68950	s/sec=2280.97 tok/sec=95060.99 u/sec=24.08	ppl=7.058026 
[2023-12-17:12:10:41:INFO:sockeye.training:__call__] E=17 B=69000	s/sec=1974.02 tok/sec=102486.29 u/sec=23.15	ppl=7.067393 
[2023-12-17:12:10:43:INFO:sockeye.training:__call__] E=17 B=69050	s/sec=2061.66 tok/sec=99706.49 u/sec=23.34	ppl=7.085312 
[2023-12-17:12:10:45:INFO:sockeye.training:__call__] E=17 B=69100	s/sec=1957.86 tok/sec=102013.53 u/sec=22.83	ppl=7.091701 
[2023-12-17:12:10:48:INFO:sockeye.training:__call__] E=17 B=69150	s/sec=2060.48 tok/sec=100859.63 u/sec=23.16	ppl=7.101484 
[2023-12-17:12:10:50:INFO:sockeye.training:__call__] E=17 B=69200	s/sec=2045.36 tok/sec=101241.46 u/sec=23.24	ppl=7.108592 
[2023-12-17:12:10:52:INFO:sockeye.training:__call__] E=17 B=69250	s/sec=1975.27 tok/sec=102728.82 u/sec=22.90	ppl=7.117612 
[2023-12-17:12:10:54:INFO:sockeye.training:__call__] E=17 B=69300	s/sec=2027.66 tok/sec=98762.57 u/sec=22.88	ppl=7.127108 
[2023-12-17:12:10:56:INFO:sockeye.training:__call__] E=17 B=69350	s/sec=2007.06 tok/sec=98293.33 u/sec=23.49	ppl=7.137926 
[2023-12-17:12:10:58:INFO:sockeye.training:__call__] E=17 B=69400	s/sec=2115.68 tok/sec=99065.49 u/sec=23.70	ppl=7.149543 
[2023-12-17:12:11:00:INFO:sockeye.training:__call__] E=17 B=69450	s/sec=1971.99 tok/sec=102879.82 u/sec=22.87	ppl=7.161744 
[2023-12-17:12:11:03:INFO:sockeye.training:__call__] E=17 B=69500	s/sec=2213.87 tok/sec=102990.02 u/sec=22.65	ppl=7.174173 
[2023-12-17:12:11:05:INFO:sockeye.training:__call__] E=17 B=69550	s/sec=2199.36 tok/sec=94643.98 u/sec=23.58	ppl=7.186319 
[2023-12-17:12:11:07:INFO:sockeye.training:__call__] E=17 B=69600	s/sec=1938.01 tok/sec=101605.58 u/sec=22.73	ppl=7.194227 
[2023-12-17:12:11:09:INFO:sockeye.training:__call__] E=17 B=69650	s/sec=2470.40 tok/sec=94349.34 u/sec=23.90	ppl=7.208120 
[2023-12-17:12:11:11:INFO:sockeye.training:__call__] E=17 B=69700	s/sec=2289.22 tok/sec=95395.45 u/sec=23.73	ppl=7.222155 
[2023-12-17:12:11:13:INFO:sockeye.training:__call__] E=17 B=69750	s/sec=2032.32 tok/sec=100477.44 u/sec=23.57	ppl=7.232366 
[2023-12-17:12:11:15:INFO:sockeye.training:__call__] E=17 B=69800	s/sec=1862.63 tok/sec=101888.63 u/sec=23.33	ppl=7.239018 
[2023-12-17:12:11:18:INFO:sockeye.training:__call__] E=17 B=69850	s/sec=2077.24 tok/sec=97840.50 u/sec=23.91	ppl=7.247012 
[2023-12-17:12:11:20:INFO:sockeye.training:__call__] E=17 B=69900	s/sec=1994.95 tok/sec=100495.34 u/sec=23.13	ppl=7.255303 
[2023-12-17:12:11:22:INFO:sockeye.training:__call__] E=17 B=69950	s/sec=2082.90 tok/sec=99564.80 u/sec=23.12	ppl=7.261468 
[2023-12-17:12:11:24:INFO:sockeye.training:__call__] E=17 B=70000	s/sec=1959.30 tok/sec=101002.92 u/sec=22.55	ppl=7.267182 
[2023-12-17:12:11:26:INFO:sockeye.training:__call__] E=17 B=70050	s/sec=2261.36 tok/sec=96085.38 u/sec=23.40	ppl=7.280473 
[2023-12-17:12:11:28:INFO:sockeye.training:__call__] E=17 B=70100	s/sec=2163.95 tok/sec=98973.93 u/sec=23.04	ppl=7.289965 
[2023-12-17:12:11:31:INFO:sockeye.training:__call__] E=17 B=70150	s/sec=2126.19 tok/sec=98721.24 u/sec=23.48	ppl=7.301354 
[2023-12-17:12:11:33:INFO:sockeye.training:__call__] E=17 B=70200	s/sec=2332.77 tok/sec=97091.67 u/sec=23.63	ppl=7.312308 
[2023-12-17:12:11:35:INFO:sockeye.training:__call__] E=17 B=70250	s/sec=2243.41 tok/sec=94555.42 u/sec=24.13	ppl=7.323669 
[2023-12-17:12:11:37:INFO:sockeye.training:__call__] E=17 B=70300	s/sec=1802.46 tok/sec=104600.56 u/sec=22.71	ppl=7.329419 
[2023-12-17:12:11:39:INFO:sockeye.training:__call__] E=17 B=70350	s/sec=2109.64 tok/sec=99515.64 u/sec=23.13	ppl=7.339555 
[2023-12-17:12:11:41:INFO:sockeye.training:__call__] E=17 B=70400	s/sec=2046.35 tok/sec=100256.41 u/sec=23.34	ppl=7.346139 
[2023-12-17:12:11:43:INFO:sockeye.training:__call__] E=17 B=70450	s/sec=1944.31 tok/sec=104018.99 u/sec=22.22	ppl=7.350725 
[2023-12-17:12:11:46:INFO:sockeye.training:__call__] E=17 B=70500	s/sec=1914.73 tok/sec=101528.87 u/sec=23.06	ppl=7.356161 
[2023-12-17:12:11:48:INFO:sockeye.training:__call__] E=17 B=70550	s/sec=2240.91 tok/sec=95662.52 u/sec=23.54	ppl=7.366147 
[2023-12-17:12:11:50:INFO:sockeye.training:__call__] E=17 B=70600	s/sec=1817.74 tok/sec=102552.93 u/sec=22.81	ppl=7.372710 
[2023-12-17:12:11:52:INFO:sockeye.training:__call__] E=17 B=70650	s/sec=2019.47 tok/sec=98576.33 u/sec=22.91	ppl=7.379375 
[2023-12-17:12:11:54:INFO:sockeye.training:__call__] E=17 B=70700	s/sec=2237.87 tok/sec=97676.69 u/sec=23.27	ppl=7.390947 
[2023-12-17:12:11:56:INFO:sockeye.training:__call__] E=17 B=70750	s/sec=2097.17 tok/sec=98013.69 u/sec=23.53	ppl=7.397752 
[2023-12-17:12:11:59:INFO:sockeye.training:__call__] E=17 B=70800	s/sec=2147.62 tok/sec=98575.42 u/sec=23.18	ppl=7.407284 
[2023-12-17:12:12:01:INFO:sockeye.training:__call__] E=17 B=70850	s/sec=1983.97 tok/sec=101912.06 u/sec=22.79	ppl=7.414583 
[2023-12-17:12:12:03:INFO:sockeye.training:__call__] E=17 B=70900	s/sec=1785.00 tok/sec=103941.16 u/sec=22.54	ppl=7.419855 
[2023-12-17:12:12:05:INFO:sockeye.training:__call__] E=17 B=70950	s/sec=2157.92 tok/sec=97462.29 u/sec=23.66	ppl=7.428399 
[2023-12-17:12:12:07:INFO:sockeye.training:__call__] E=17 B=71000	s/sec=2011.67 tok/sec=101051.96 u/sec=23.24	ppl=7.435676 
[2023-12-17:12:12:09:INFO:sockeye.training:__call__] E=17 B=71050	s/sec=2210.32 tok/sec=97663.50 u/sec=23.70	ppl=7.443737 
[2023-12-17:12:12:11:INFO:sockeye.training:__call__] E=17 B=71100	s/sec=2370.30 tok/sec=94318.63 u/sec=24.09	ppl=7.450865 
[2023-12-17:12:12:14:INFO:sockeye.training:__call__] E=17 B=71150	s/sec=1868.83 tok/sec=102363.38 u/sec=22.95	ppl=7.456211 
[2023-12-17:12:12:16:INFO:sockeye.training:__call__] E=17 B=71200	s/sec=2122.81 tok/sec=99828.36 u/sec=23.40	ppl=7.464293 
[2023-12-17:12:12:18:INFO:sockeye.training:__call__] E=17 B=71250	s/sec=2259.43 tok/sec=99116.76 u/sec=23.30	ppl=7.473204 
[2023-12-17:12:12:20:INFO:sockeye.training:__call__] E=17 B=71300	s/sec=2025.19 tok/sec=98462.05 u/sec=23.31	ppl=7.479381 
[2023-12-17:12:12:22:INFO:sockeye.training:__call__] E=17 B=71350	s/sec=2049.45 tok/sec=99966.21 u/sec=23.04	ppl=7.485516 
[2023-12-17:12:12:24:INFO:sockeye.training:__call__] E=17 B=71400	s/sec=2214.91 tok/sec=96927.64 u/sec=23.58	ppl=7.493100 
[2023-12-17:12:12:26:INFO:sockeye.training:__call__] E=17 B=71450	s/sec=2203.86 tok/sec=98899.22 u/sec=23.19	ppl=7.500657 
[2023-12-17:12:12:29:INFO:sockeye.training:__call__] E=17 B=71500	s/sec=1827.60 tok/sec=103711.54 u/sec=22.71	ppl=7.505533 
[2023-12-17:12:12:31:INFO:sockeye.training:__call__] E=17 B=71550	s/sec=1733.37 tok/sec=103616.13 u/sec=22.34	ppl=7.509906 
[2023-12-17:12:12:33:INFO:sockeye.training:__call__] E=17 B=71600	s/sec=2033.00 tok/sec=99728.92 u/sec=23.27	ppl=7.516208 
[2023-12-17:12:12:35:INFO:sockeye.training:__call__] E=17 B=71650	s/sec=2494.98 tok/sec=94127.26 u/sec=24.21	ppl=7.526026 
[2023-12-17:12:12:37:INFO:sockeye.training:__call__] E=17 B=71700	s/sec=2146.80 tok/sec=97402.23 u/sec=23.58	ppl=7.531691 
[2023-12-17:12:12:39:INFO:sockeye.training:__call__] E=17 B=71750	s/sec=2092.12 tok/sec=96818.06 u/sec=23.27	ppl=7.539629 
[2023-12-17:12:12:41:INFO:sockeye.training:__call__] E=17 B=71800	s/sec=2225.05 tok/sec=95592.66 u/sec=24.06	ppl=7.546406 
[2023-12-17:12:12:44:INFO:sockeye.training:__call__] E=17 B=71850	s/sec=1935.58 tok/sec=101684.40 u/sec=23.09	ppl=7.550109 
[2023-12-17:12:12:46:INFO:sockeye.training:__call__] E=17 B=71900	s/sec=1971.28 tok/sec=98903.21 u/sec=23.56	ppl=7.555808 
[2023-12-17:12:12:48:INFO:sockeye.training:__call__] E=18 B=71950	s/sec=1799.28 tok/sec=96339.08 u/sec=21.58	ppl=7.546675 
[2023-12-17:12:12:50:INFO:sockeye.training:__call__] E=18 B=72000	s/sec=1935.21 tok/sec=102446.24 u/sec=22.69	ppl=7.533156 
[2023-12-17:12:12:50:INFO:sockeye.training:_create_checkpoint] Checkpoint [18]	Updates=72000 Epoch=18 Samples=6416504 Time-cost=172.222 Updates/sec=23.226
[2023-12-17:12:12:50:INFO:sockeye.training:_create_checkpoint] Checkpoint [18]	Train-ppl=7.533156
[2023-12-17:12:12:53:INFO:sockeye.training:_evaluate] Checkpoint [18]	Validation-ppl=30.135231	Validation-bleu=0.177819	Validation-chrf=31.900580	Validation-rouge1=0.164104	Validation-rouge2=0.079882	Validation-rougel=0.155898	Validation-length-ratio-mse=1.277313	Validation-ter=95.130004	Validation-avg-sec-per-sent=0.003229	Validation-decode-walltime=1.614295
[2023-12-17:12:12:53:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 13 checkpoints, best so far: 31.983236
[2023-12-17:12:12:53:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [18]	Learning-rate=0.000180
[2023-12-17:12:12:53:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:12:53:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=5, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:12:54:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00018"
[2023-12-17:12:12:56:INFO:sockeye.training:__call__] E=18 B=72050	s/sec=780.72 tok/sec=40220.61 u/sec=8.79	ppl=6.655512 
[2023-12-17:12:12:58:INFO:sockeye.training:__call__] E=18 B=72100	s/sec=1879.04 tok/sec=103578.41 u/sec=22.58	ppl=6.656816 
[2023-12-17:12:13:00:INFO:sockeye.training:__call__] E=18 B=72150	s/sec=1704.86 tok/sec=102644.35 u/sec=22.96	ppl=6.608070 
[2023-12-17:12:13:03:INFO:sockeye.training:__call__] E=18 B=72200	s/sec=2120.33 tok/sec=100523.82 u/sec=23.29	ppl=6.653016 
[2023-12-17:12:13:05:INFO:sockeye.training:__call__] E=18 B=72250	s/sec=1914.67 tok/sec=102174.73 u/sec=23.06	ppl=6.663231 
[2023-12-17:12:13:07:INFO:sockeye.training:__call__] E=18 B=72300	s/sec=1918.98 tok/sec=99075.74 u/sec=22.67	ppl=6.685006 
[2023-12-17:12:13:09:INFO:sockeye.training:__call__] E=18 B=72350	s/sec=2173.09 tok/sec=98572.53 u/sec=23.10	ppl=6.709171 
[2023-12-17:12:13:11:INFO:sockeye.training:__call__] E=18 B=72400	s/sec=2095.30 tok/sec=95789.51 u/sec=23.81	ppl=6.727787 
[2023-12-17:12:13:13:INFO:sockeye.training:__call__] E=18 B=72450	s/sec=2057.83 tok/sec=97490.44 u/sec=23.91	ppl=6.745930 
[2023-12-17:12:13:15:INFO:sockeye.training:__call__] E=18 B=72500	s/sec=2010.68 tok/sec=95446.20 u/sec=23.80	ppl=6.755170 
[2023-12-17:12:13:18:INFO:sockeye.training:__call__] E=18 B=72550	s/sec=2098.03 tok/sec=101386.50 u/sec=23.13	ppl=6.767388 
[2023-12-17:12:13:20:INFO:sockeye.training:__call__] E=18 B=72600	s/sec=1901.93 tok/sec=102902.32 u/sec=22.90	ppl=6.768543 
[2023-12-17:12:13:22:INFO:sockeye.training:__call__] E=18 B=72650	s/sec=2229.19 tok/sec=95142.61 u/sec=23.94	ppl=6.781203 
[2023-12-17:12:13:24:INFO:sockeye.training:__call__] E=18 B=72700	s/sec=2170.94 tok/sec=95653.68 u/sec=22.96	ppl=6.794539 
[2023-12-17:12:13:26:INFO:sockeye.training:__call__] E=18 B=72750	s/sec=2093.71 tok/sec=98679.96 u/sec=23.28	ppl=6.802872 
[2023-12-17:12:13:28:INFO:sockeye.training:__call__] E=18 B=72800	s/sec=2142.46 tok/sec=99961.69 u/sec=23.17	ppl=6.814816 
[2023-12-17:12:13:30:INFO:sockeye.training:__call__] E=18 B=72850	s/sec=1982.16 tok/sec=99841.06 u/sec=23.37	ppl=6.822539 
[2023-12-17:12:13:33:INFO:sockeye.training:__call__] E=18 B=72900	s/sec=2078.85 tok/sec=100954.11 u/sec=22.83	ppl=6.833322 
[2023-12-17:12:13:35:INFO:sockeye.training:__call__] E=18 B=72950	s/sec=2140.11 tok/sec=98887.15 u/sec=23.42	ppl=6.841570 
[2023-12-17:12:13:37:INFO:sockeye.training:__call__] E=18 B=73000	s/sec=1777.74 tok/sec=104865.30 u/sec=22.31	ppl=6.844534 
[2023-12-17:12:13:39:INFO:sockeye.training:__call__] E=18 B=73050	s/sec=2151.17 tok/sec=98004.47 u/sec=23.18	ppl=6.861860 
[2023-12-17:12:13:41:INFO:sockeye.training:__call__] E=18 B=73100	s/sec=1901.66 tok/sec=99218.28 u/sec=23.58	ppl=6.860430 
[2023-12-17:12:13:43:INFO:sockeye.training:__call__] E=18 B=73150	s/sec=2102.72 tok/sec=96318.13 u/sec=23.55	ppl=6.867967 
[2023-12-17:12:13:46:INFO:sockeye.training:__call__] E=18 B=73200	s/sec=2113.50 tok/sec=97571.60 u/sec=23.30	ppl=6.882972 
[2023-12-17:12:13:48:INFO:sockeye.training:__call__] E=18 B=73250	s/sec=2353.46 tok/sec=94711.72 u/sec=23.57	ppl=6.901840 
[2023-12-17:12:13:50:INFO:sockeye.training:__call__] E=18 B=73300	s/sec=2078.14 tok/sec=95955.85 u/sec=23.53	ppl=6.908409 
[2023-12-17:12:13:52:INFO:sockeye.training:__call__] E=18 B=73350	s/sec=2239.86 tok/sec=96077.18 u/sec=23.89	ppl=6.926531 
[2023-12-17:12:13:54:INFO:sockeye.training:__call__] E=18 B=73400	s/sec=2154.38 tok/sec=98498.60 u/sec=23.26	ppl=6.937321 
[2023-12-17:12:13:56:INFO:sockeye.training:__call__] E=18 B=73450	s/sec=1973.21 tok/sec=101852.24 u/sec=22.67	ppl=6.947029 
[2023-12-17:12:13:58:INFO:sockeye.training:__call__] E=18 B=73500	s/sec=2074.36 tok/sec=95130.57 u/sec=22.67	ppl=6.956728 
[2023-12-17:12:14:01:INFO:sockeye.training:__call__] E=18 B=73550	s/sec=2007.64 tok/sec=100972.51 u/sec=22.77	ppl=6.961990 
[2023-12-17:12:14:03:INFO:sockeye.training:__call__] E=18 B=73600	s/sec=2148.01 tok/sec=98062.30 u/sec=22.64	ppl=6.971984 
[2023-12-17:12:14:05:INFO:sockeye.training:__call__] E=18 B=73650	s/sec=2129.30 tok/sec=100639.16 u/sec=23.10	ppl=6.980356 
[2023-12-17:12:14:07:INFO:sockeye.training:__call__] E=18 B=73700	s/sec=2134.06 tok/sec=96409.52 u/sec=23.78	ppl=6.989750 
[2023-12-17:12:14:09:INFO:sockeye.training:__call__] E=18 B=73750	s/sec=1972.13 tok/sec=99123.12 u/sec=23.39	ppl=6.997135 
[2023-12-17:12:14:11:INFO:sockeye.training:__call__] E=18 B=73800	s/sec=2260.33 tok/sec=95819.74 u/sec=23.86	ppl=7.008366 
[2023-12-17:12:14:14:INFO:sockeye.training:__call__] E=18 B=73850	s/sec=2006.85 tok/sec=108534.93 u/sec=21.48	ppl=7.017158 
[2023-12-17:12:14:16:INFO:sockeye.training:__call__] E=18 B=73900	s/sec=2002.94 tok/sec=102322.11 u/sec=22.89	ppl=7.024320 
[2023-12-17:12:14:18:INFO:sockeye.training:__call__] E=18 B=73950	s/sec=2016.18 tok/sec=102152.96 u/sec=23.08	ppl=7.031170 
[2023-12-17:12:14:20:INFO:sockeye.training:__call__] E=18 B=74000	s/sec=1790.03 tok/sec=103988.95 u/sec=22.69	ppl=7.036245 
[2023-12-17:12:14:22:INFO:sockeye.training:__call__] E=18 B=74050	s/sec=1906.08 tok/sec=103743.64 u/sec=22.95	ppl=7.043629 
[2023-12-17:12:14:25:INFO:sockeye.training:__call__] E=18 B=74100	s/sec=1997.51 tok/sec=99778.95 u/sec=23.25	ppl=7.051516 
[2023-12-17:12:14:27:INFO:sockeye.training:__call__] E=18 B=74150	s/sec=2022.70 tok/sec=101105.08 u/sec=23.24	ppl=7.056507 
[2023-12-17:12:14:29:INFO:sockeye.training:__call__] E=18 B=74200	s/sec=1944.75 tok/sec=101918.18 u/sec=23.33	ppl=7.062079 
[2023-12-17:12:14:31:INFO:sockeye.training:__call__] E=18 B=74250	s/sec=2282.48 tok/sec=95380.86 u/sec=23.98	ppl=7.072396 
[2023-12-17:12:14:33:INFO:sockeye.training:__call__] E=18 B=74300	s/sec=2031.71 tok/sec=98357.33 u/sec=23.30	ppl=7.079931 
[2023-12-17:12:14:35:INFO:sockeye.training:__call__] E=18 B=74350	s/sec=2085.85 tok/sec=98299.95 u/sec=23.32	ppl=7.089007 
[2023-12-17:12:14:37:INFO:sockeye.training:__call__] E=18 B=74400	s/sec=2125.92 tok/sec=100147.96 u/sec=23.15	ppl=7.099747 
[2023-12-17:12:14:40:INFO:sockeye.training:__call__] E=18 B=74450	s/sec=1981.12 tok/sec=101839.24 u/sec=23.32	ppl=7.103636 
[2023-12-17:12:14:42:INFO:sockeye.training:__call__] E=18 B=74500	s/sec=2027.55 tok/sec=98588.85 u/sec=23.55	ppl=7.111374 
[2023-12-17:12:14:44:INFO:sockeye.training:__call__] E=18 B=74550	s/sec=2069.18 tok/sec=99501.06 u/sec=23.47	ppl=7.118152 
[2023-12-17:12:14:46:INFO:sockeye.training:__call__] E=18 B=74600	s/sec=2139.28 tok/sec=98399.24 u/sec=23.42	ppl=7.127467 
[2023-12-17:12:14:48:INFO:sockeye.training:__call__] E=18 B=74650	s/sec=2049.83 tok/sec=98119.95 u/sec=23.59	ppl=7.135586 
[2023-12-17:12:14:50:INFO:sockeye.training:__call__] E=18 B=74700	s/sec=2313.52 tok/sec=96321.72 u/sec=23.82	ppl=7.143660 
[2023-12-17:12:14:52:INFO:sockeye.training:__call__] E=18 B=74750	s/sec=1977.21 tok/sec=102164.47 u/sec=22.59	ppl=7.149853 
[2023-12-17:12:14:54:INFO:sockeye.training:__call__] E=18 B=74800	s/sec=2131.87 tok/sec=99277.90 u/sec=23.38	ppl=7.157881 
[2023-12-17:12:14:57:INFO:sockeye.training:__call__] E=18 B=74850	s/sec=1994.77 tok/sec=97210.52 u/sec=22.83	ppl=7.165036 
[2023-12-17:12:14:59:INFO:sockeye.training:__call__] E=18 B=74900	s/sec=2159.78 tok/sec=91995.00 u/sec=22.80	ppl=7.174223 
[2023-12-17:12:15:01:INFO:sockeye.training:__call__] E=18 B=74950	s/sec=2075.39 tok/sec=97710.84 u/sec=23.29	ppl=7.183889 
[2023-12-17:12:15:03:INFO:sockeye.training:__call__] E=18 B=75000	s/sec=1917.91 tok/sec=102551.33 u/sec=22.83	ppl=7.189054 
[2023-12-17:12:15:05:INFO:sockeye.training:__call__] E=18 B=75050	s/sec=2233.38 tok/sec=97121.91 u/sec=23.58	ppl=7.195109 
[2023-12-17:12:15:07:INFO:sockeye.training:__call__] E=18 B=75100	s/sec=2172.62 tok/sec=98432.84 u/sec=23.21	ppl=7.204277 
[2023-12-17:12:15:10:INFO:sockeye.training:__call__] E=18 B=75150	s/sec=2177.74 tok/sec=96177.15 u/sec=23.23	ppl=7.213835 
[2023-12-17:12:15:12:INFO:sockeye.training:__call__] E=18 B=75200	s/sec=1865.75 tok/sec=95712.49 u/sec=22.69	ppl=7.221176 
[2023-12-17:12:15:14:INFO:sockeye.training:__call__] E=18 B=75250	s/sec=2284.32 tok/sec=93765.35 u/sec=23.37	ppl=7.232192 
[2023-12-17:12:15:16:INFO:sockeye.training:__call__] E=18 B=75300	s/sec=2040.98 tok/sec=101670.06 u/sec=22.78	ppl=7.238938 
[2023-12-17:12:15:18:INFO:sockeye.training:__call__] E=18 B=75350	s/sec=2213.97 tok/sec=98701.00 u/sec=23.37	ppl=7.245905 
[2023-12-17:12:15:21:INFO:sockeye.training:__call__] E=18 B=75400	s/sec=1848.63 tok/sec=102622.75 u/sec=22.70	ppl=7.251069 
[2023-12-17:12:15:23:INFO:sockeye.training:__call__] E=18 B=75450	s/sec=2292.96 tok/sec=95607.59 u/sec=23.92	ppl=7.258980 
[2023-12-17:12:15:25:INFO:sockeye.training:__call__] E=18 B=75500	s/sec=2238.71 tok/sec=97953.08 u/sec=23.88	ppl=7.267207 
[2023-12-17:12:15:27:INFO:sockeye.training:__call__] E=18 B=75550	s/sec=2157.97 tok/sec=98603.79 u/sec=23.02	ppl=7.276090 
[2023-12-17:12:15:29:INFO:sockeye.training:__call__] E=18 B=75600	s/sec=1970.74 tok/sec=97986.78 u/sec=23.24	ppl=7.280593 
[2023-12-17:12:15:31:INFO:sockeye.training:__call__] E=18 B=75650	s/sec=2042.35 tok/sec=96133.14 u/sec=22.83	ppl=7.287321 
[2023-12-17:12:15:33:INFO:sockeye.training:__call__] E=18 B=75700	s/sec=1957.03 tok/sec=103421.90 u/sec=22.12	ppl=7.291122 
[2023-12-17:12:15:36:INFO:sockeye.training:__call__] E=18 B=75750	s/sec=2199.21 tok/sec=96233.28 u/sec=23.74	ppl=7.297548 
[2023-12-17:12:15:38:INFO:sockeye.training:__call__] E=18 B=75800	s/sec=2136.73 tok/sec=100210.54 u/sec=23.27	ppl=7.305508 
[2023-12-17:12:15:40:INFO:sockeye.training:__call__] E=18 B=75850	s/sec=2156.63 tok/sec=98622.53 u/sec=23.04	ppl=7.313422 
[2023-12-17:12:15:42:INFO:sockeye.training:__call__] E=18 B=75900	s/sec=2008.72 tok/sec=98977.84 u/sec=23.25	ppl=7.318254 
[2023-12-17:12:15:44:INFO:sockeye.training:__call__] E=19 B=75950	s/sec=1854.60 tok/sec=94955.67 u/sec=21.87	ppl=7.309117 
[2023-12-17:12:15:47:INFO:sockeye.training:__call__] E=19 B=76000	s/sec=1855.74 tok/sec=101712.20 u/sec=23.01	ppl=7.296937 
[2023-12-17:12:15:47:INFO:sockeye.training:_create_checkpoint] Checkpoint [19]	Updates=76000 Epoch=19 Samples=6772576 Time-cost=172.779 Updates/sec=23.151
[2023-12-17:12:15:47:INFO:sockeye.training:_create_checkpoint] Checkpoint [19]	Train-ppl=7.296937
[2023-12-17:12:15:49:INFO:sockeye.training:_evaluate] Checkpoint [19]	Validation-ppl=31.853540	Validation-bleu=0.161435	Validation-chrf=31.492669	Validation-rouge1=0.167680	Validation-rouge2=0.080657	Validation-rougel=0.159907	Validation-length-ratio-mse=1.288595	Validation-ter=95.749071	Validation-avg-sec-per-sent=0.003272	Validation-decode-walltime=1.636203
[2023-12-17:12:15:49:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 14 checkpoints, best so far: 31.983236
[2023-12-17:12:15:49:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [19]	Learning-rate=0.000180
[2023-12-17:12:15:50:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:15:50:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=6, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:15:50:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00019"
[2023-12-17:12:15:52:INFO:sockeye.training:__call__] E=19 B=76050	s/sec=871.22 tok/sec=36990.87 u/sec=8.85	ppl=6.569601 
[2023-12-17:12:15:54:INFO:sockeye.training:__call__] E=19 B=76100	s/sec=2155.34 tok/sec=92877.63 u/sec=23.35	ppl=6.550563 
[2023-12-17:12:15:56:INFO:sockeye.training:__call__] E=19 B=76150	s/sec=2220.98 tok/sec=98654.36 u/sec=23.10	ppl=6.521896 
[2023-12-17:12:15:59:INFO:sockeye.training:__call__] E=19 B=76200	s/sec=2056.74 tok/sec=99375.64 u/sec=23.16	ppl=6.532642 
[2023-12-17:12:16:01:INFO:sockeye.training:__call__] E=19 B=76250	s/sec=2053.36 tok/sec=99845.38 u/sec=23.33	ppl=6.537674 
[2023-12-17:12:16:03:INFO:sockeye.training:__call__] E=19 B=76300	s/sec=2070.28 tok/sec=97573.37 u/sec=23.44	ppl=6.537692 
[2023-12-17:12:16:05:INFO:sockeye.training:__call__] E=19 B=76350	s/sec=1953.09 tok/sec=97669.70 u/sec=23.93	ppl=6.548775 
[2023-12-17:12:16:07:INFO:sockeye.training:__call__] E=19 B=76400	s/sec=1787.06 tok/sec=101581.61 u/sec=23.08	ppl=6.548746 
[2023-12-17:12:16:09:INFO:sockeye.training:__call__] E=19 B=76450	s/sec=2180.11 tok/sec=97226.07 u/sec=23.37	ppl=6.565093 
[2023-12-17:12:16:12:INFO:sockeye.training:__call__] E=19 B=76500	s/sec=1921.11 tok/sec=102895.79 u/sec=22.74	ppl=6.571258 
[2023-12-17:12:16:14:INFO:sockeye.training:__call__] E=19 B=76550	s/sec=2424.18 tok/sec=93416.23 u/sec=23.71	ppl=6.597521 
[2023-12-17:12:16:16:INFO:sockeye.training:__call__] E=19 B=76600	s/sec=2179.76 tok/sec=98306.35 u/sec=23.41	ppl=6.609164 
[2023-12-17:12:16:18:INFO:sockeye.training:__call__] E=19 B=76650	s/sec=2116.19 tok/sec=95140.94 u/sec=23.83	ppl=6.622841 
[2023-12-17:12:16:20:INFO:sockeye.training:__call__] E=19 B=76700	s/sec=2237.88 tok/sec=97890.05 u/sec=23.71	ppl=6.632985 
[2023-12-17:12:16:22:INFO:sockeye.training:__call__] E=19 B=76750	s/sec=2180.30 tok/sec=96472.65 u/sec=23.41	ppl=6.648442 
[2023-12-17:12:16:24:INFO:sockeye.training:__call__] E=19 B=76800	s/sec=1904.20 tok/sec=99957.23 u/sec=23.29	ppl=6.655141 
[2023-12-17:12:16:26:INFO:sockeye.training:__call__] E=19 B=76850	s/sec=2088.76 tok/sec=97147.99 u/sec=23.61	ppl=6.664548 
[2023-12-17:12:16:28:INFO:sockeye.training:__call__] E=19 B=76900	s/sec=2209.52 tok/sec=97142.97 u/sec=23.77	ppl=6.677757 
[2023-12-17:12:16:31:INFO:sockeye.training:__call__] E=19 B=76950	s/sec=1838.94 tok/sec=101278.75 u/sec=22.54	ppl=6.681769 
[2023-12-17:12:16:33:INFO:sockeye.training:__call__] E=19 B=77000	s/sec=2216.78 tok/sec=98745.36 u/sec=23.29	ppl=6.688867 
[2023-12-17:12:16:35:INFO:sockeye.training:__call__] E=19 B=77050	s/sec=1909.70 tok/sec=101927.74 u/sec=22.52	ppl=6.690539 
[2023-12-17:12:16:37:INFO:sockeye.training:__call__] E=19 B=77100	s/sec=2076.57 tok/sec=95514.65 u/sec=23.38	ppl=6.703369 
[2023-12-17:12:16:39:INFO:sockeye.training:__call__] E=19 B=77150	s/sec=1968.03 tok/sec=100881.78 u/sec=23.21	ppl=6.707748 
[2023-12-17:12:16:41:INFO:sockeye.training:__call__] E=19 B=77200	s/sec=2138.25 tok/sec=94422.85 u/sec=24.12	ppl=6.716942 
[2023-12-17:12:16:44:INFO:sockeye.training:__call__] E=19 B=77250	s/sec=2095.33 tok/sec=97522.66 u/sec=23.39	ppl=6.725604 
[2023-12-17:12:16:46:INFO:sockeye.training:__call__] E=19 B=77300	s/sec=2187.98 tok/sec=97899.94 u/sec=23.78	ppl=6.736841 
[2023-12-17:12:16:48:INFO:sockeye.training:__call__] E=19 B=77350	s/sec=2183.08 tok/sec=98366.02 u/sec=23.44	ppl=6.747660 
[2023-12-17:12:16:50:INFO:sockeye.training:__call__] E=19 B=77400	s/sec=2033.18 tok/sec=103622.46 u/sec=22.98	ppl=6.751646 
[2023-12-17:12:16:52:INFO:sockeye.training:__call__] E=19 B=77450	s/sec=2089.82 tok/sec=99494.96 u/sec=23.62	ppl=6.762566 
[2023-12-17:12:16:54:INFO:sockeye.training:__call__] E=19 B=77500	s/sec=2101.17 tok/sec=100111.11 u/sec=23.28	ppl=6.772397 
[2023-12-17:12:16:56:INFO:sockeye.training:__call__] E=19 B=77550	s/sec=2040.25 tok/sec=101815.57 u/sec=22.81	ppl=6.779693 
[2023-12-17:12:16:59:INFO:sockeye.training:__call__] E=19 B=77600	s/sec=2151.17 tok/sec=98467.34 u/sec=23.38	ppl=6.787984 
[2023-12-17:12:17:01:INFO:sockeye.training:__call__] E=19 B=77650	s/sec=2030.52 tok/sec=101206.90 u/sec=23.20	ppl=6.795412 
[2023-12-17:12:17:03:INFO:sockeye.training:__call__] E=19 B=77700	s/sec=2163.39 tok/sec=98668.62 u/sec=23.64	ppl=6.803980 
[2023-12-17:12:17:05:INFO:sockeye.training:__call__] E=19 B=77750	s/sec=1868.70 tok/sec=101674.91 u/sec=22.77	ppl=6.808807 
[2023-12-17:12:17:07:INFO:sockeye.training:__call__] E=19 B=77800	s/sec=1992.46 tok/sec=97465.86 u/sec=23.41	ppl=6.815867 
[2023-12-17:12:17:09:INFO:sockeye.training:__call__] E=19 B=77850	s/sec=2113.34 tok/sec=96832.09 u/sec=23.84	ppl=6.825154 
[2023-12-17:12:17:11:INFO:sockeye.training:__call__] E=19 B=77900	s/sec=2143.75 tok/sec=97681.59 u/sec=23.84	ppl=6.833995 
[2023-12-17:12:17:13:INFO:sockeye.training:__call__] E=19 B=77950	s/sec=2193.94 tok/sec=97752.74 u/sec=23.52	ppl=6.844472 
[2023-12-17:12:17:16:INFO:sockeye.training:__call__] E=19 B=78000	s/sec=1916.81 tok/sec=102820.87 u/sec=23.08	ppl=6.851425 
[2023-12-17:12:17:18:INFO:sockeye.training:__call__] E=19 B=78050	s/sec=1996.10 tok/sec=103840.69 u/sec=22.56	ppl=6.857741 
[2023-12-17:12:17:20:INFO:sockeye.training:__call__] E=19 B=78100	s/sec=1957.84 tok/sec=101350.43 u/sec=23.04	ppl=6.866029 
[2023-12-17:12:17:22:INFO:sockeye.training:__call__] E=19 B=78150	s/sec=2005.24 tok/sec=98875.67 u/sec=23.30	ppl=6.872853 
[2023-12-17:12:17:24:INFO:sockeye.training:__call__] E=19 B=78200	s/sec=2111.57 tok/sec=99162.33 u/sec=23.15	ppl=6.880788 
[2023-12-17:12:17:26:INFO:sockeye.training:__call__] E=19 B=78250	s/sec=2040.49 tok/sec=98213.94 u/sec=23.49	ppl=6.889473 
[2023-12-17:12:17:29:INFO:sockeye.training:__call__] E=19 B=78300	s/sec=1779.57 tok/sec=104524.82 u/sec=22.24	ppl=6.895321 
[2023-12-17:12:17:31:INFO:sockeye.training:__call__] E=19 B=78350	s/sec=2191.09 tok/sec=99269.77 u/sec=23.29	ppl=6.905513 
[2023-12-17:12:17:33:INFO:sockeye.training:__call__] E=19 B=78400	s/sec=2004.07 tok/sec=101707.46 u/sec=23.15	ppl=6.910663 
[2023-12-17:12:17:35:INFO:sockeye.training:__call__] E=19 B=78450	s/sec=2184.55 tok/sec=97244.40 u/sec=23.42	ppl=6.919477 
[2023-12-17:12:17:37:INFO:sockeye.training:__call__] E=19 B=78500	s/sec=2121.43 tok/sec=98664.93 u/sec=23.10	ppl=6.927117 
[2023-12-17:12:17:39:INFO:sockeye.training:__call__] E=19 B=78550	s/sec=2099.57 tok/sec=97673.29 u/sec=23.56	ppl=6.933558 
[2023-12-17:12:17:42:INFO:sockeye.training:__call__] E=19 B=78600	s/sec=2076.05 tok/sec=97484.26 u/sec=23.51	ppl=6.942225 
[2023-12-17:12:17:44:INFO:sockeye.training:__call__] E=19 B=78650	s/sec=1936.15 tok/sec=101048.46 u/sec=23.05	ppl=6.945034 
[2023-12-17:12:17:46:INFO:sockeye.training:__call__] E=19 B=78700	s/sec=2195.18 tok/sec=94070.38 u/sec=23.99	ppl=6.953214 
[2023-12-17:12:17:48:INFO:sockeye.training:__call__] E=19 B=78750	s/sec=2218.54 tok/sec=98853.90 u/sec=23.03	ppl=6.962781 
[2023-12-17:12:17:50:INFO:sockeye.training:__call__] E=19 B=78800	s/sec=2186.40 tok/sec=97203.23 u/sec=23.36	ppl=6.972182 
[2023-12-17:12:17:52:INFO:sockeye.training:__call__] E=19 B=78850	s/sec=2039.03 tok/sec=101387.81 u/sec=22.88	ppl=6.979170 
[2023-12-17:12:17:54:INFO:sockeye.training:__call__] E=19 B=78900	s/sec=2205.08 tok/sec=99448.21 u/sec=23.24	ppl=6.987292 
[2023-12-17:12:17:57:INFO:sockeye.training:__call__] E=19 B=78950	s/sec=1909.42 tok/sec=101086.93 u/sec=23.40	ppl=6.995012 
[2023-12-17:12:17:59:INFO:sockeye.training:__call__] E=19 B=79000	s/sec=2369.11 tok/sec=93851.59 u/sec=23.77	ppl=7.004389 
[2023-12-17:12:18:01:INFO:sockeye.training:__call__] E=19 B=79050	s/sec=1999.80 tok/sec=101710.71 u/sec=23.28	ppl=7.010500 
[2023-12-17:12:18:03:INFO:sockeye.training:__call__] E=19 B=79100	s/sec=2076.70 tok/sec=101428.82 u/sec=23.14	ppl=7.018921 
[2023-12-17:12:18:05:INFO:sockeye.training:__call__] E=19 B=79150	s/sec=1921.84 tok/sec=101616.78 u/sec=22.45	ppl=7.024153 
[2023-12-17:12:18:07:INFO:sockeye.training:__call__] E=19 B=79200	s/sec=2032.98 tok/sec=100293.90 u/sec=23.02	ppl=7.030531 
[2023-12-17:12:18:10:INFO:sockeye.training:__call__] E=19 B=79250	s/sec=1938.14 tok/sec=103855.76 u/sec=22.43	ppl=7.034799 
[2023-12-17:12:18:12:INFO:sockeye.training:__call__] E=19 B=79300	s/sec=2181.14 tok/sec=104893.21 u/sec=22.13	ppl=7.041060 
[2023-12-17:12:18:14:INFO:sockeye.training:__call__] E=19 B=79350	s/sec=2028.57 tok/sec=99626.34 u/sec=23.18	ppl=7.048179 
[2023-12-17:12:18:16:INFO:sockeye.training:__call__] E=19 B=79400	s/sec=1915.09 tok/sec=101954.93 u/sec=22.58	ppl=7.052150 
[2023-12-17:12:18:19:INFO:sockeye.training:__call__] E=19 B=79450	s/sec=1934.92 tok/sec=103486.64 u/sec=21.99	ppl=7.056580 
[2023-12-17:12:18:21:INFO:sockeye.training:__call__] E=19 B=79500	s/sec=1922.59 tok/sec=103602.18 u/sec=22.67	ppl=7.061821 
[2023-12-17:12:18:23:INFO:sockeye.training:__call__] E=19 B=79550	s/sec=2200.67 tok/sec=99090.44 u/sec=23.55	ppl=7.069725 
[2023-12-17:12:18:25:INFO:sockeye.training:__call__] E=19 B=79600	s/sec=1918.02 tok/sec=105433.19 u/sec=22.70	ppl=7.074300 
[2023-12-17:12:18:27:INFO:sockeye.training:__call__] E=19 B=79650	s/sec=2087.01 tok/sec=100652.75 u/sec=23.25	ppl=7.078458 
[2023-12-17:12:18:29:INFO:sockeye.training:__call__] E=19 B=79700	s/sec=2059.35 tok/sec=99299.01 u/sec=23.44	ppl=7.083422 
[2023-12-17:12:18:31:INFO:sockeye.training:__call__] E=19 B=79750	s/sec=2290.75 tok/sec=95880.21 u/sec=24.06	ppl=7.091527 
[2023-12-17:12:18:34:INFO:sockeye.training:__call__] E=19 B=79800	s/sec=2262.48 tok/sec=94716.23 u/sec=24.05	ppl=7.097999 
[2023-12-17:12:18:36:INFO:sockeye.training:__call__] E=19 B=79850	s/sec=2221.61 tok/sec=99032.99 u/sec=23.30	ppl=7.106331 
[2023-12-17:12:18:38:INFO:sockeye.training:__call__] E=19 B=79900	s/sec=1875.05 tok/sec=102103.15 u/sec=23.11	ppl=7.113401 
[2023-12-17:12:18:40:INFO:sockeye.training:__call__] E=20 B=79950	s/sec=1857.21 tok/sec=97921.68 u/sec=21.70	ppl=7.101489 
[2023-12-17:12:18:42:INFO:sockeye.training:__call__] E=20 B=80000	s/sec=2243.26 tok/sec=97391.29 u/sec=23.56	ppl=7.092047 
[2023-12-17:12:18:42:INFO:sockeye.training:_create_checkpoint] Checkpoint [20]	Updates=80000 Epoch=20 Samples=7129840 Time-cost=172.315 Updates/sec=23.213
[2023-12-17:12:18:42:INFO:sockeye.training:_create_checkpoint] Checkpoint [20]	Train-ppl=7.092047
[2023-12-17:12:18:45:INFO:sockeye.training:_evaluate] Checkpoint [20]	Validation-ppl=32.059560	Validation-bleu=0.158444	Validation-chrf=30.095456	Validation-rouge1=0.156664	Validation-rouge2=0.074008	Validation-rougel=0.149906	Validation-length-ratio-mse=1.275953	Validation-ter=95.955427	Validation-avg-sec-per-sent=0.003679	Validation-decode-walltime=1.839712
[2023-12-17:12:18:45:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 15 checkpoints, best so far: 31.983236
[2023-12-17:12:18:45:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 31.983236 > 0.000000 over 20 checkpoints
[2023-12-17:12:18:45:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [20]	Learning-rate=0.000180
[2023-12-17:12:18:46:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:18:46:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=7, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:18:46:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00020"
[2023-12-17:12:18:48:INFO:sockeye.training:__call__] E=20 B=80050	s/sec=760.85 tok/sec=38022.91 u/sec=8.76	ppl=6.285661 
[2023-12-17:12:18:50:INFO:sockeye.training:__call__] E=20 B=80100	s/sec=1933.72 tok/sec=100612.24 u/sec=22.98	ppl=6.313898 
[2023-12-17:12:18:52:INFO:sockeye.training:__call__] E=20 B=80150	s/sec=2122.89 tok/sec=96459.42 u/sec=23.61	ppl=6.343372 
[2023-12-17:12:18:54:INFO:sockeye.training:__call__] E=20 B=80200	s/sec=2361.44 tok/sec=97031.03 u/sec=23.65	ppl=6.376526 
[2023-12-17:12:18:57:INFO:sockeye.training:__call__] E=20 B=80250	s/sec=1997.03 tok/sec=100751.22 u/sec=22.49	ppl=6.379919 
[2023-12-17:12:18:59:INFO:sockeye.training:__call__] E=20 B=80300	s/sec=1982.44 tok/sec=100016.99 u/sec=22.73	ppl=6.396636 
[2023-12-17:12:19:01:INFO:sockeye.training:__call__] E=20 B=80350	s/sec=1775.81 tok/sec=103177.84 u/sec=22.60	ppl=6.395059 
[2023-12-17:12:19:03:INFO:sockeye.training:__call__] E=20 B=80400	s/sec=2162.08 tok/sec=96493.73 u/sec=23.92	ppl=6.413126 
[2023-12-17:12:19:05:INFO:sockeye.training:__call__] E=20 B=80450	s/sec=2150.84 tok/sec=99103.73 u/sec=23.42	ppl=6.426313 
[2023-12-17:12:19:07:INFO:sockeye.training:__call__] E=20 B=80500	s/sec=2246.97 tok/sec=98313.15 u/sec=23.37	ppl=6.439665 
[2023-12-17:12:19:10:INFO:sockeye.training:__call__] E=20 B=80550	s/sec=1986.08 tok/sec=99379.45 u/sec=23.60	ppl=6.449165 
[2023-12-17:12:19:12:INFO:sockeye.training:__call__] E=20 B=80600	s/sec=2207.10 tok/sec=99040.67 u/sec=23.54	ppl=6.464504 
[2023-12-17:12:19:14:INFO:sockeye.training:__call__] E=20 B=80650	s/sec=1948.32 tok/sec=100041.29 u/sec=23.37	ppl=6.472466 
[2023-12-17:12:19:16:INFO:sockeye.training:__call__] E=20 B=80700	s/sec=2064.52 tok/sec=100438.35 u/sec=23.29	ppl=6.479078 
[2023-12-17:12:19:18:INFO:sockeye.training:__call__] E=20 B=80750	s/sec=2056.12 tok/sec=101493.83 u/sec=22.79	ppl=6.481815 
[2023-12-17:12:19:20:INFO:sockeye.training:__call__] E=20 B=80800	s/sec=1968.67 tok/sec=100350.31 u/sec=23.22	ppl=6.484483 
[2023-12-17:12:19:22:INFO:sockeye.training:__call__] E=20 B=80850	s/sec=2035.10 tok/sec=97203.23 u/sec=23.51	ppl=6.494877 
[2023-12-17:12:19:25:INFO:sockeye.training:__call__] E=20 B=80900	s/sec=2253.11 tok/sec=97352.44 u/sec=23.63	ppl=6.513594 
[2023-12-17:12:19:27:INFO:sockeye.training:__call__] E=20 B=80950	s/sec=2176.81 tok/sec=98020.22 u/sec=23.50	ppl=6.521518 
[2023-12-17:12:19:29:INFO:sockeye.training:__call__] E=20 B=81000	s/sec=2078.98 tok/sec=98862.20 u/sec=23.41	ppl=6.530746 
[2023-12-17:12:19:31:INFO:sockeye.training:__call__] E=20 B=81050	s/sec=2170.44 tok/sec=97741.41 u/sec=23.51	ppl=6.541063 
[2023-12-17:12:19:33:INFO:sockeye.training:__call__] E=20 B=81100	s/sec=2189.11 tok/sec=96701.48 u/sec=23.79	ppl=6.555166 
[2023-12-17:12:19:35:INFO:sockeye.training:__call__] E=20 B=81150	s/sec=2139.92 tok/sec=101859.98 u/sec=23.06	ppl=6.563903 
[2023-12-17:12:19:37:INFO:sockeye.training:__call__] E=20 B=81200	s/sec=2094.53 tok/sec=99758.18 u/sec=23.42	ppl=6.569619 
[2023-12-17:12:19:40:INFO:sockeye.training:__call__] E=20 B=81250	s/sec=2191.60 tok/sec=97054.33 u/sec=23.22	ppl=6.582074 
[2023-12-17:12:19:42:INFO:sockeye.training:__call__] E=20 B=81300	s/sec=2184.32 tok/sec=98242.03 u/sec=23.46	ppl=6.591335 
[2023-12-17:12:19:44:INFO:sockeye.training:__call__] E=20 B=81350	s/sec=2297.76 tok/sec=95191.63 u/sec=23.74	ppl=6.607169 
[2023-12-17:12:19:46:INFO:sockeye.training:__call__] E=20 B=81400	s/sec=2050.95 tok/sec=98713.77 u/sec=23.26	ppl=6.613617 
[2023-12-17:12:19:48:INFO:sockeye.training:__call__] E=20 B=81450	s/sec=2037.34 tok/sec=97914.08 u/sec=23.76	ppl=6.620028 
[2023-12-17:12:19:50:INFO:sockeye.training:__call__] E=20 B=81500	s/sec=2127.56 tok/sec=102210.54 u/sec=22.89	ppl=6.628505 
[2023-12-17:12:19:52:INFO:sockeye.training:__call__] E=20 B=81550	s/sec=1932.03 tok/sec=99147.54 u/sec=23.40	ppl=6.634301 
[2023-12-17:12:19:55:INFO:sockeye.training:__call__] E=20 B=81600	s/sec=2038.30 tok/sec=101708.32 u/sec=23.08	ppl=6.639507 
[2023-12-17:12:19:57:INFO:sockeye.training:__call__] E=20 B=81650	s/sec=1758.89 tok/sec=103587.82 u/sec=22.85	ppl=6.647408 
[2023-12-17:12:19:59:INFO:sockeye.training:__call__] E=20 B=81700	s/sec=2143.49 tok/sec=96826.39 u/sec=23.80	ppl=6.657153 
[2023-12-17:12:20:01:INFO:sockeye.training:__call__] E=20 B=81750	s/sec=2020.64 tok/sec=103188.07 u/sec=22.84	ppl=6.661757 
[2023-12-17:12:20:03:INFO:sockeye.training:__call__] E=20 B=81800	s/sec=2259.08 tok/sec=94873.82 u/sec=24.14	ppl=6.673292 
[2023-12-17:12:20:05:INFO:sockeye.training:__call__] E=20 B=81850	s/sec=2259.39 tok/sec=94105.11 u/sec=23.50	ppl=6.682556 
[2023-12-17:12:20:07:INFO:sockeye.training:__call__] E=20 B=81900	s/sec=2096.56 tok/sec=104538.02 u/sec=22.10	ppl=6.690091 
[2023-12-17:12:20:10:INFO:sockeye.training:__call__] E=20 B=81950	s/sec=2214.91 tok/sec=96968.95 u/sec=23.79	ppl=6.697361 
[2023-12-17:12:20:12:INFO:sockeye.training:__call__] E=20 B=82000	s/sec=2000.30 tok/sec=98231.87 u/sec=23.41	ppl=6.705483 
[2023-12-17:12:20:14:INFO:sockeye.training:__call__] E=20 B=82050	s/sec=1843.68 tok/sec=100827.44 u/sec=23.05	ppl=6.711568 
[2023-12-17:12:20:16:INFO:sockeye.training:__call__] E=20 B=82100	s/sec=2028.68 tok/sec=101075.79 u/sec=23.31	ppl=6.719622 
[2023-12-17:12:20:18:INFO:sockeye.training:__call__] E=20 B=82150	s/sec=2117.88 tok/sec=100251.47 u/sec=23.39	ppl=6.727101 
[2023-12-17:12:20:20:INFO:sockeye.training:__call__] E=20 B=82200	s/sec=2047.89 tok/sec=98388.78 u/sec=23.48	ppl=6.733320 
[2023-12-17:12:20:22:INFO:sockeye.training:__call__] E=20 B=82250	s/sec=2040.51 tok/sec=99541.33 u/sec=23.10	ppl=6.741435 
[2023-12-17:12:20:25:INFO:sockeye.training:__call__] E=20 B=82300	s/sec=1961.49 tok/sec=101214.52 u/sec=23.09	ppl=6.748946 
[2023-12-17:12:20:27:INFO:sockeye.training:__call__] E=20 B=82350	s/sec=2117.04 tok/sec=99245.09 u/sec=23.17	ppl=6.758100 
[2023-12-17:12:20:29:INFO:sockeye.training:__call__] E=20 B=82400	s/sec=2282.65 tok/sec=95579.43 u/sec=24.02	ppl=6.767465 
[2023-12-17:12:20:31:INFO:sockeye.training:__call__] E=20 B=82450	s/sec=2060.14 tok/sec=99898.20 u/sec=23.24	ppl=6.773712 
[2023-12-17:12:20:33:INFO:sockeye.training:__call__] E=20 B=82500	s/sec=2017.66 tok/sec=100428.34 u/sec=22.93	ppl=6.778262 
[2023-12-17:12:20:35:INFO:sockeye.training:__call__] E=20 B=82550	s/sec=2114.71 tok/sec=97451.71 u/sec=23.81	ppl=6.783685 
[2023-12-17:12:20:37:INFO:sockeye.training:__call__] E=20 B=82600	s/sec=2055.61 tok/sec=101217.70 u/sec=23.15	ppl=6.789914 
[2023-12-17:12:20:40:INFO:sockeye.training:__call__] E=20 B=82650	s/sec=1865.72 tok/sec=103165.97 u/sec=22.82	ppl=6.795774 
[2023-12-17:12:20:42:INFO:sockeye.training:__call__] E=20 B=82700	s/sec=1969.50 tok/sec=104379.61 u/sec=22.75	ppl=6.802006 
[2023-12-17:12:20:44:INFO:sockeye.training:__call__] E=20 B=82750	s/sec=2140.74 tok/sec=97738.08 u/sec=23.47	ppl=6.809593 
[2023-12-17:12:20:46:INFO:sockeye.training:__call__] E=20 B=82800	s/sec=2043.28 tok/sec=100624.16 u/sec=23.13	ppl=6.813652 
[2023-12-17:12:20:48:INFO:sockeye.training:__call__] E=20 B=82850	s/sec=2005.40 tok/sec=102699.62 u/sec=22.66	ppl=6.818660 
[2023-12-17:12:20:50:INFO:sockeye.training:__call__] E=20 B=82900	s/sec=2059.45 tok/sec=97330.73 u/sec=23.79	ppl=6.824484 
[2023-12-17:12:20:53:INFO:sockeye.training:__call__] E=20 B=82950	s/sec=2032.26 tok/sec=101714.14 u/sec=22.60	ppl=6.831115 
[2023-12-17:12:20:55:INFO:sockeye.training:__call__] E=20 B=83000	s/sec=2161.86 tok/sec=97342.71 u/sec=23.14	ppl=6.838728 
[2023-12-17:12:20:57:INFO:sockeye.training:__call__] E=20 B=83050	s/sec=2347.81 tok/sec=97892.85 u/sec=23.04	ppl=6.846215 
[2023-12-17:12:20:59:INFO:sockeye.training:__call__] E=20 B=83100	s/sec=2079.68 tok/sec=95543.05 u/sec=23.89	ppl=6.852022 
[2023-12-17:12:21:01:INFO:sockeye.training:__call__] E=20 B=83150	s/sec=2075.99 tok/sec=98129.65 u/sec=23.25	ppl=6.858401 
[2023-12-17:12:21:03:INFO:sockeye.training:__call__] E=20 B=83200	s/sec=2180.53 tok/sec=96337.12 u/sec=23.66	ppl=6.865254 
[2023-12-17:12:21:06:INFO:sockeye.training:__call__] E=20 B=83250	s/sec=1984.92 tok/sec=99791.85 u/sec=23.06	ppl=6.871397 
[2023-12-17:12:21:08:INFO:sockeye.training:__call__] E=20 B=83300	s/sec=1902.28 tok/sec=103443.56 u/sec=22.73	ppl=6.873577 
[2023-12-17:12:21:10:INFO:sockeye.training:__call__] E=20 B=83350	s/sec=2125.88 tok/sec=101895.82 u/sec=22.91	ppl=6.879540 
[2023-12-17:12:21:12:INFO:sockeye.training:__call__] E=20 B=83400	s/sec=2043.16 tok/sec=99047.59 u/sec=23.82	ppl=6.885876 
[2023-12-17:12:21:14:INFO:sockeye.training:__call__] E=20 B=83450	s/sec=2154.41 tok/sec=98968.62 u/sec=23.30	ppl=6.891840 
[2023-12-17:12:21:16:INFO:sockeye.training:__call__] E=20 B=83500	s/sec=1854.33 tok/sec=101262.57 u/sec=22.90	ppl=6.897275 
[2023-12-17:12:21:18:INFO:sockeye.training:__call__] E=20 B=83550	s/sec=2131.91 tok/sec=95950.96 u/sec=23.62	ppl=6.904540 
[2023-12-17:12:21:21:INFO:sockeye.training:__call__] E=20 B=83600	s/sec=2014.12 tok/sec=100326.46 u/sec=22.80	ppl=6.910046 
[2023-12-17:12:21:23:INFO:sockeye.training:__call__] E=20 B=83650	s/sec=2069.05 tok/sec=98346.28 u/sec=23.64	ppl=6.915024 
[2023-12-17:12:21:25:INFO:sockeye.training:__call__] E=20 B=83700	s/sec=1970.74 tok/sec=102257.63 u/sec=23.02	ppl=6.919572 
[2023-12-17:12:21:27:INFO:sockeye.training:__call__] E=20 B=83750	s/sec=2166.13 tok/sec=98484.69 u/sec=23.18	ppl=6.926217 
[2023-12-17:12:21:29:INFO:sockeye.training:__call__] E=20 B=83800	s/sec=1876.30 tok/sec=102449.44 u/sec=22.77	ppl=6.931338 
[2023-12-17:12:21:31:INFO:sockeye.training:__call__] E=20 B=83850	s/sec=1950.69 tok/sec=103706.79 u/sec=22.75	ppl=6.935958 
[2023-12-17:12:21:34:INFO:sockeye.training:__call__] E=21 B=83900	s/sec=1883.50 tok/sec=95198.52 u/sec=21.76	ppl=6.940595 
[2023-12-17:12:21:36:INFO:sockeye.training:__call__] E=21 B=83950	s/sec=2144.56 tok/sec=97507.03 u/sec=23.19	ppl=6.931458 
[2023-12-17:12:21:38:INFO:sockeye.training:__call__] E=21 B=84000	s/sec=1968.90 tok/sec=102672.59 u/sec=22.62	ppl=6.922625 
[2023-12-17:12:21:38:INFO:sockeye.training:_create_checkpoint] Checkpoint [21]	Updates=84000 Epoch=21 Samples=7486136 Time-cost=172.291 Updates/sec=23.216
[2023-12-17:12:21:38:INFO:sockeye.training:_create_checkpoint] Checkpoint [21]	Train-ppl=6.922625
[2023-12-17:12:21:41:INFO:sockeye.training:_evaluate] Checkpoint [21]	Validation-ppl=32.811089	Validation-bleu=0.149172	Validation-chrf=28.994237	Validation-rouge1=0.152784	Validation-rouge2=0.074439	Validation-rougel=0.145765	Validation-length-ratio-mse=1.345232	Validation-ter=100.247627	Validation-avg-sec-per-sent=0.003303	Validation-decode-walltime=1.651370
[2023-12-17:12:21:41:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 16 checkpoints, best so far: 31.983236
[2023-12-17:12:21:41:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 3.215049 > 0.000000 over 20 checkpoints
[2023-12-17:12:21:41:INFO:sockeye.lr_scheduler:new_evaluation_result] 8 checkpoints since improvement or rate scaling, lowering learning rate: 1.80e-04 -> 1.62e-04
[2023-12-17:12:21:41:INFO:sockeye.training:_adjust_learning_rate] Loading model parameters and optimizer states from best checkpoint: 5
[2023-12-17:12:21:41:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best" to "cuda:0"
[2023-12-17:12:21:41:INFO:sockeye.training:_load_optimizer_state] Loaded optimizer state from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:12:21:41:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [21]	Learning-rate=0.000180
[2023-12-17:12:21:42:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:21:42:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:21:42:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00021"
[2023-12-17:12:21:44:INFO:sockeye.training:__call__] E=21 B=84050	s/sec=728.75 tok/sec=35500.36 u/sec=8.30	ppl=8.235107 
[2023-12-17:12:21:46:INFO:sockeye.training:__call__] E=21 B=84100	s/sec=2229.26 tok/sec=97271.49 u/sec=23.62	ppl=8.250975 
[2023-12-17:12:21:48:INFO:sockeye.training:__call__] E=21 B=84150	s/sec=2106.77 tok/sec=100279.42 u/sec=22.78	ppl=8.255132 
[2023-12-17:12:21:51:INFO:sockeye.training:__call__] E=21 B=84200	s/sec=2086.60 tok/sec=99467.32 u/sec=23.41	ppl=8.263311 
[2023-12-17:12:21:53:INFO:sockeye.training:__call__] E=21 B=84250	s/sec=2077.08 tok/sec=98981.19 u/sec=23.39	ppl=8.273792 
[2023-12-17:12:21:55:INFO:sockeye.training:__call__] E=21 B=84300	s/sec=1935.33 tok/sec=102056.82 u/sec=22.87	ppl=8.252656 
[2023-12-17:12:21:57:INFO:sockeye.training:__call__] E=21 B=84350	s/sec=2112.87 tok/sec=101506.80 u/sec=23.25	ppl=8.263791 
[2023-12-17:12:21:59:INFO:sockeye.training:__call__] E=21 B=84400	s/sec=2162.21 tok/sec=97182.16 u/sec=23.67	ppl=8.292101 
[2023-12-17:12:22:01:INFO:sockeye.training:__call__] E=21 B=84450	s/sec=1734.71 tok/sec=104991.52 u/sec=22.54	ppl=8.272024 
[2023-12-17:12:22:04:INFO:sockeye.training:__call__] E=21 B=84500	s/sec=2061.76 tok/sec=98781.67 u/sec=23.82	ppl=8.280892 
[2023-12-17:12:22:06:INFO:sockeye.training:__call__] E=21 B=84550	s/sec=2342.67 tok/sec=96697.99 u/sec=23.85	ppl=8.300305 
[2023-12-17:12:22:08:INFO:sockeye.training:__call__] E=21 B=84600	s/sec=2045.07 tok/sec=100850.66 u/sec=23.07	ppl=8.304572 
[2023-12-17:12:22:10:INFO:sockeye.training:__call__] E=21 B=84650	s/sec=2101.95 tok/sec=97806.45 u/sec=23.42	ppl=8.306284 
[2023-12-17:12:22:12:INFO:sockeye.training:__call__] E=21 B=84700	s/sec=2104.69 tok/sec=97197.01 u/sec=23.36	ppl=8.311434 
[2023-12-17:12:22:14:INFO:sockeye.training:__call__] E=21 B=84750	s/sec=2152.57 tok/sec=97110.76 u/sec=23.56	ppl=8.331409 
[2023-12-17:12:22:16:INFO:sockeye.training:__call__] E=21 B=84800	s/sec=1963.11 tok/sec=99857.11 u/sec=23.02	ppl=8.337479 
[2023-12-17:12:22:18:INFO:sockeye.training:__call__] E=21 B=84850	s/sec=2135.63 tok/sec=97448.12 u/sec=23.54	ppl=8.356623 
[2023-12-17:12:22:21:INFO:sockeye.training:__call__] E=21 B=84900	s/sec=2080.20 tok/sec=97493.97 u/sec=23.90	ppl=8.376492 
[2023-12-17:12:22:23:INFO:sockeye.training:__call__] E=21 B=84950	s/sec=1902.13 tok/sec=101664.21 u/sec=23.13	ppl=8.382380 
[2023-12-17:12:22:25:INFO:sockeye.training:__call__] E=21 B=85000	s/sec=2092.42 tok/sec=99206.06 u/sec=23.44	ppl=8.386780 
[2023-12-17:12:22:27:INFO:sockeye.training:__call__] E=21 B=85050	s/sec=1924.81 tok/sec=102203.58 u/sec=23.09	ppl=8.386099 
[2023-12-17:12:22:29:INFO:sockeye.training:__call__] E=21 B=85100	s/sec=1874.97 tok/sec=104006.38 u/sec=22.80	ppl=8.388428 
[2023-12-17:12:22:31:INFO:sockeye.training:__call__] E=21 B=85150	s/sec=2272.69 tok/sec=95993.59 u/sec=23.79	ppl=8.400283 
[2023-12-17:12:22:33:INFO:sockeye.training:__call__] E=21 B=85200	s/sec=1901.15 tok/sec=101364.81 u/sec=23.16	ppl=8.405705 
[2023-12-17:12:22:36:INFO:sockeye.training:__call__] E=21 B=85250	s/sec=2112.28 tok/sec=98487.34 u/sec=23.62	ppl=8.410861 
[2023-12-17:12:22:38:INFO:sockeye.training:__call__] E=21 B=85300	s/sec=1976.16 tok/sec=98843.98 u/sec=22.58	ppl=8.421157 
[2023-12-17:12:22:40:INFO:sockeye.training:__call__] E=21 B=85350	s/sec=1928.50 tok/sec=103244.58 u/sec=22.91	ppl=8.426752 
[2023-12-17:12:22:42:INFO:sockeye.training:__call__] E=21 B=85400	s/sec=2142.95 tok/sec=96904.96 u/sec=23.62	ppl=8.442187 
[2023-12-17:12:22:44:INFO:sockeye.training:__call__] E=21 B=85450	s/sec=2308.88 tok/sec=98031.84 u/sec=23.46	ppl=8.457267 
[2023-12-17:12:22:46:INFO:sockeye.training:__call__] E=21 B=85500	s/sec=1954.11 tok/sec=100120.93 u/sec=22.87	ppl=8.469481 
[2023-12-17:12:22:49:INFO:sockeye.training:__call__] E=21 B=85550	s/sec=1752.07 tok/sec=113921.19 u/sec=20.86	ppl=8.472761 
[2023-12-17:12:22:51:INFO:sockeye.training:__call__] E=21 B=85600	s/sec=2153.62 tok/sec=98769.52 u/sec=23.21	ppl=8.480860 
[2023-12-17:12:22:53:INFO:sockeye.training:__call__] E=21 B=85650	s/sec=2411.78 tok/sec=96592.41 u/sec=23.66	ppl=8.497657 
[2023-12-17:12:22:55:INFO:sockeye.training:__call__] E=21 B=85700	s/sec=2230.54 tok/sec=100047.13 u/sec=23.23	ppl=8.518990 
[2023-12-17:12:22:57:INFO:sockeye.training:__call__] E=21 B=85750	s/sec=2318.70 tok/sec=95728.20 u/sec=23.76	ppl=8.535432 
[2023-12-17:12:22:59:INFO:sockeye.training:__call__] E=21 B=85800	s/sec=2099.77 tok/sec=98389.31 u/sec=23.43	ppl=8.546487 
[2023-12-17:12:23:02:INFO:sockeye.training:__call__] E=21 B=85850	s/sec=1954.07 tok/sec=100911.55 u/sec=22.79	ppl=8.551546 
[2023-12-17:12:23:04:INFO:sockeye.training:__call__] E=21 B=85900	s/sec=2019.44 tok/sec=101960.13 u/sec=22.70	ppl=8.559439 
[2023-12-17:12:23:06:INFO:sockeye.training:__call__] E=21 B=85950	s/sec=2293.59 tok/sec=97515.41 u/sec=23.73	ppl=8.567747 
[2023-12-17:12:23:08:INFO:sockeye.training:__call__] E=21 B=86000	s/sec=2163.87 tok/sec=96843.55 u/sec=23.85	ppl=8.575915 
[2023-12-17:12:23:10:INFO:sockeye.training:__call__] E=21 B=86050	s/sec=2168.57 tok/sec=98010.22 u/sec=23.29	ppl=8.585305 
[2023-12-17:12:23:12:INFO:sockeye.training:__call__] E=21 B=86100	s/sec=2209.45 tok/sec=97860.99 u/sec=23.44	ppl=8.596834 
[2023-12-17:12:23:14:INFO:sockeye.training:__call__] E=21 B=86150	s/sec=2288.49 tok/sec=98114.67 u/sec=23.56	ppl=8.607284 
[2023-12-17:12:23:17:INFO:sockeye.training:__call__] E=21 B=86200	s/sec=2096.51 tok/sec=98562.01 u/sec=23.44	ppl=8.618154 
[2023-12-17:12:23:19:INFO:sockeye.training:__call__] E=21 B=86250	s/sec=2139.05 tok/sec=98911.60 u/sec=23.33	ppl=8.628411 
[2023-12-17:12:23:21:INFO:sockeye.training:__call__] E=21 B=86300	s/sec=1977.63 tok/sec=100393.75 u/sec=22.72	ppl=8.632874 
[2023-12-17:12:23:23:INFO:sockeye.training:__call__] E=21 B=86350	s/sec=1949.89 tok/sec=100254.50 u/sec=23.12	ppl=8.637703 
[2023-12-17:12:23:25:INFO:sockeye.training:__call__] E=21 B=86400	s/sec=1990.72 tok/sec=100468.39 u/sec=23.13	ppl=8.639429 
[2023-12-17:12:23:27:INFO:sockeye.training:__call__] E=21 B=86450	s/sec=2113.70 tok/sec=97237.87 u/sec=23.72	ppl=8.651461 
[2023-12-17:12:23:30:INFO:sockeye.training:__call__] E=21 B=86500	s/sec=1829.98 tok/sec=101226.54 u/sec=23.29	ppl=8.656353 
[2023-12-17:12:23:32:INFO:sockeye.training:__call__] E=21 B=86550	s/sec=2268.07 tok/sec=94627.96 u/sec=23.47	ppl=8.667628 
[2023-12-17:12:23:34:INFO:sockeye.training:__call__] E=21 B=86600	s/sec=2140.13 tok/sec=97188.80 u/sec=23.38	ppl=8.676264 
[2023-12-17:12:23:36:INFO:sockeye.training:__call__] E=21 B=86650	s/sec=2101.69 tok/sec=100350.36 u/sec=23.04	ppl=8.684607 
[2023-12-17:12:23:38:INFO:sockeye.training:__call__] E=21 B=86700	s/sec=2327.68 tok/sec=95701.35 u/sec=24.13	ppl=8.699639 
[2023-12-17:12:23:40:INFO:sockeye.training:__call__] E=21 B=86750	s/sec=2424.98 tok/sec=93137.66 u/sec=23.76	ppl=8.708366 
[2023-12-17:12:23:42:INFO:sockeye.training:__call__] E=21 B=86800	s/sec=2222.33 tok/sec=97377.91 u/sec=23.42	ppl=8.716655 
[2023-12-17:12:23:44:INFO:sockeye.training:__call__] E=21 B=86850	s/sec=2100.33 tok/sec=98158.18 u/sec=23.07	ppl=8.723593 
[2023-12-17:12:23:47:INFO:sockeye.training:__call__] E=21 B=86900	s/sec=1976.44 tok/sec=99932.29 u/sec=23.13	ppl=8.729459 
[2023-12-17:12:23:49:INFO:sockeye.training:__call__] E=21 B=86950	s/sec=1830.71 tok/sec=102316.68 u/sec=22.66	ppl=8.735782 
[2023-12-17:12:23:51:INFO:sockeye.training:__call__] E=21 B=87000	s/sec=2146.02 tok/sec=94905.30 u/sec=23.45	ppl=8.742850 
[2023-12-17:12:23:53:INFO:sockeye.training:__call__] E=21 B=87050	s/sec=2036.21 tok/sec=97947.90 u/sec=23.39	ppl=8.749377 
[2023-12-17:12:23:55:INFO:sockeye.training:__call__] E=21 B=87100	s/sec=1923.42 tok/sec=102777.95 u/sec=22.94	ppl=8.755489 
[2023-12-17:12:23:57:INFO:sockeye.training:__call__] E=21 B=87150	s/sec=2147.56 tok/sec=97245.16 u/sec=23.63	ppl=8.760519 
[2023-12-17:12:24:00:INFO:sockeye.training:__call__] E=21 B=87200	s/sec=1935.62 tok/sec=100338.93 u/sec=23.31	ppl=8.765941 
[2023-12-17:12:24:02:INFO:sockeye.training:__call__] E=21 B=87250	s/sec=1764.73 tok/sec=102982.46 u/sec=22.56	ppl=8.769765 
[2023-12-17:12:24:04:INFO:sockeye.training:__call__] E=21 B=87300	s/sec=2072.62 tok/sec=100197.81 u/sec=23.17	ppl=8.775374 
[2023-12-17:12:24:06:INFO:sockeye.training:__call__] E=21 B=87350	s/sec=2144.82 tok/sec=98252.61 u/sec=23.68	ppl=8.782972 
[2023-12-17:12:24:08:INFO:sockeye.training:__call__] E=21 B=87400	s/sec=2055.07 tok/sec=101432.39 u/sec=22.94	ppl=8.789146 
[2023-12-17:12:24:10:INFO:sockeye.training:__call__] E=21 B=87450	s/sec=1990.22 tok/sec=99570.07 u/sec=23.03	ppl=8.794666 
[2023-12-17:12:24:13:INFO:sockeye.training:__call__] E=21 B=87500	s/sec=2037.96 tok/sec=99143.86 u/sec=22.95	ppl=8.803141 
[2023-12-17:12:24:15:INFO:sockeye.training:__call__] E=21 B=87550	s/sec=2072.22 tok/sec=97541.14 u/sec=23.51	ppl=8.807621 
[2023-12-17:12:24:17:INFO:sockeye.training:__call__] E=21 B=87600	s/sec=2226.58 tok/sec=96022.91 u/sec=23.95	ppl=8.814576 
[2023-12-17:12:24:19:INFO:sockeye.training:__call__] E=21 B=87650	s/sec=1910.89 tok/sec=99752.75 u/sec=23.24	ppl=8.821069 
[2023-12-17:12:24:21:INFO:sockeye.training:__call__] E=21 B=87700	s/sec=2080.84 tok/sec=98689.33 u/sec=23.27	ppl=8.828689 
[2023-12-17:12:24:23:INFO:sockeye.training:__call__] E=21 B=87750	s/sec=2157.19 tok/sec=99870.57 u/sec=23.53	ppl=8.837337 
[2023-12-17:12:24:25:INFO:sockeye.training:__call__] E=21 B=87800	s/sec=1985.96 tok/sec=103298.62 u/sec=22.57	ppl=8.845083 
[2023-12-17:12:24:28:INFO:sockeye.training:__call__] E=21 B=87850	s/sec=1757.68 tok/sec=103699.52 u/sec=22.74	ppl=8.845849 
[2023-12-17:12:24:30:INFO:sockeye.training:__call__] E=22 B=87900	s/sec=1887.97 tok/sec=96641.51 u/sec=22.18	ppl=8.847503 
[2023-12-17:12:24:32:INFO:sockeye.training:__call__] E=22 B=87950	s/sec=1851.62 tok/sec=104470.30 u/sec=22.56	ppl=8.827910 
[2023-12-17:12:24:34:INFO:sockeye.training:__call__] E=22 B=88000	s/sec=2004.53 tok/sec=98240.54 u/sec=23.24	ppl=8.811137 
[2023-12-17:12:24:34:INFO:sockeye.training:_create_checkpoint] Checkpoint [22]	Updates=88000 Epoch=22 Samples=7842232 Time-cost=172.242 Updates/sec=23.223
[2023-12-17:12:24:34:INFO:sockeye.training:_create_checkpoint] Checkpoint [22]	Train-ppl=8.811137
[2023-12-17:12:24:37:INFO:sockeye.training:_evaluate] Checkpoint [22]	Validation-ppl=27.261654	Validation-bleu=0.153659	Validation-chrf=30.401315	Validation-rouge1=0.164726	Validation-rouge2=0.077968	Validation-rougel=0.158489	Validation-length-ratio-mse=1.258889	Validation-ter=95.253818	Validation-avg-sec-per-sent=0.003145	Validation-decode-walltime=1.572691
[2023-12-17:12:24:37:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 17 checkpoints, best so far: 31.983236
[2023-12-17:12:24:37:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 1.383042 > 0.000000 over 20 checkpoints
[2023-12-17:12:24:37:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [22]	Learning-rate=0.000162
[2023-12-17:12:24:37:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:24:37:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:24:38:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00022"
[2023-12-17:12:24:40:INFO:sockeye.training:__call__] E=22 B=88050	s/sec=850.44 tok/sec=36089.55 u/sec=8.99	ppl=7.690584 
[2023-12-17:12:24:42:INFO:sockeye.training:__call__] E=22 B=88100	s/sec=2042.30 tok/sec=99409.59 u/sec=23.42	ppl=7.743049 
[2023-12-17:12:24:44:INFO:sockeye.training:__call__] E=22 B=88150	s/sec=2087.54 tok/sec=99963.59 u/sec=23.26	ppl=7.687191 
[2023-12-17:12:24:46:INFO:sockeye.training:__call__] E=22 B=88200	s/sec=2155.93 tok/sec=97655.63 u/sec=23.35	ppl=7.681806 
[2023-12-17:12:24:48:INFO:sockeye.training:__call__] E=22 B=88250	s/sec=1798.29 tok/sec=100233.37 u/sec=22.98	ppl=7.666700 
[2023-12-17:12:24:51:INFO:sockeye.training:__call__] E=22 B=88300	s/sec=2082.08 tok/sec=98042.26 u/sec=23.53	ppl=7.698421 
[2023-12-17:12:24:53:INFO:sockeye.training:__call__] E=22 B=88350	s/sec=2045.59 tok/sec=100560.11 u/sec=23.16	ppl=7.704035 
[2023-12-17:12:24:55:INFO:sockeye.training:__call__] E=22 B=88400	s/sec=1812.00 tok/sec=103039.98 u/sec=23.11	ppl=7.720610 
[2023-12-17:12:24:57:INFO:sockeye.training:__call__] E=22 B=88450	s/sec=1786.02 tok/sec=105091.94 u/sec=22.15	ppl=7.739639 
[2023-12-17:12:24:59:INFO:sockeye.training:__call__] E=22 B=88500	s/sec=1993.93 tok/sec=101122.34 u/sec=22.62	ppl=7.764221 
[2023-12-17:12:25:01:INFO:sockeye.training:__call__] E=22 B=88550	s/sec=1944.88 tok/sec=99746.64 u/sec=23.56	ppl=7.771814 
[2023-12-17:12:25:04:INFO:sockeye.training:__call__] E=22 B=88600	s/sec=2058.19 tok/sec=97956.24 u/sec=23.35	ppl=7.778841 
[2023-12-17:12:25:06:INFO:sockeye.training:__call__] E=22 B=88650	s/sec=1991.03 tok/sec=101013.85 u/sec=22.63	ppl=7.785477 
[2023-12-17:12:25:08:INFO:sockeye.training:__call__] E=22 B=88700	s/sec=2173.83 tok/sec=101743.72 u/sec=22.57	ppl=7.796485 
[2023-12-17:12:25:10:INFO:sockeye.training:__call__] E=22 B=88750	s/sec=2029.52 tok/sec=99420.54 u/sec=23.02	ppl=7.822881 
[2023-12-17:12:25:12:INFO:sockeye.training:__call__] E=22 B=88800	s/sec=2026.37 tok/sec=97205.38 u/sec=23.50	ppl=7.839229 
[2023-12-17:12:25:14:INFO:sockeye.training:__call__] E=22 B=88850	s/sec=2124.02 tok/sec=98096.39 u/sec=23.54	ppl=7.855832 
[2023-12-17:12:25:17:INFO:sockeye.training:__call__] E=22 B=88900	s/sec=1925.65 tok/sec=101427.31 u/sec=22.97	ppl=7.867652 
[2023-12-17:12:25:19:INFO:sockeye.training:__call__] E=22 B=88950	s/sec=2078.53 tok/sec=96793.33 u/sec=23.97	ppl=7.881715 
[2023-12-17:12:25:21:INFO:sockeye.training:__call__] E=22 B=89000	s/sec=2161.70 tok/sec=99468.01 u/sec=23.29	ppl=7.899899 
[2023-12-17:12:25:23:INFO:sockeye.training:__call__] E=22 B=89050	s/sec=2086.55 tok/sec=97985.11 u/sec=23.54	ppl=7.914262 
[2023-12-17:12:25:25:INFO:sockeye.training:__call__] E=22 B=89100	s/sec=1966.89 tok/sec=101713.00 u/sec=22.64	ppl=7.920670 
[2023-12-17:12:25:27:INFO:sockeye.training:__call__] E=22 B=89150	s/sec=2153.95 tok/sec=100522.73 u/sec=23.09	ppl=7.930490 
[2023-12-17:12:25:30:INFO:sockeye.training:__call__] E=22 B=89200	s/sec=1821.35 tok/sec=102314.94 u/sec=22.68	ppl=7.943069 
[2023-12-17:12:25:32:INFO:sockeye.training:__call__] E=22 B=89250	s/sec=2194.23 tok/sec=96864.34 u/sec=23.69	ppl=7.952993 
[2023-12-17:12:25:34:INFO:sockeye.training:__call__] E=22 B=89300	s/sec=1973.18 tok/sec=101604.08 u/sec=23.01	ppl=7.962889 
[2023-12-17:12:25:36:INFO:sockeye.training:__call__] E=22 B=89350	s/sec=2002.07 tok/sec=99740.61 u/sec=23.13	ppl=7.973974 
[2023-12-17:12:25:38:INFO:sockeye.training:__call__] E=22 B=89400	s/sec=2105.90 tok/sec=99051.87 u/sec=23.34	ppl=7.979236 
[2023-12-17:12:25:40:INFO:sockeye.training:__call__] E=22 B=89450	s/sec=2052.68 tok/sec=100621.31 u/sec=22.71	ppl=7.985352 
[2023-12-17:12:25:43:INFO:sockeye.training:__call__] E=22 B=89500	s/sec=1839.24 tok/sec=102880.43 u/sec=22.85	ppl=7.985802 
[2023-12-17:12:25:45:INFO:sockeye.training:__call__] E=22 B=89550	s/sec=2187.75 tok/sec=98460.15 u/sec=23.37	ppl=7.997873 
[2023-12-17:12:25:47:INFO:sockeye.training:__call__] E=22 B=89600	s/sec=2266.63 tok/sec=94577.04 u/sec=23.69	ppl=8.010507 
[2023-12-17:12:25:49:INFO:sockeye.training:__call__] E=22 B=89650	s/sec=1981.10 tok/sec=101414.58 u/sec=22.97	ppl=8.019476 
[2023-12-17:12:25:51:INFO:sockeye.training:__call__] E=22 B=89700	s/sec=2217.68 tok/sec=96663.51 u/sec=23.82	ppl=8.026947 
[2023-12-17:12:25:53:INFO:sockeye.training:__call__] E=22 B=89750	s/sec=2348.38 tok/sec=95505.98 u/sec=23.60	ppl=8.039519 
[2023-12-17:12:25:55:INFO:sockeye.training:__call__] E=22 B=89800	s/sec=1900.19 tok/sec=101778.93 u/sec=23.20	ppl=8.040463 
[2023-12-17:12:25:58:INFO:sockeye.training:__call__] E=22 B=89850	s/sec=2054.62 tok/sec=99865.78 u/sec=23.14	ppl=8.050833 
[2023-12-17:12:26:00:INFO:sockeye.training:__call__] E=22 B=89900	s/sec=2248.63 tok/sec=96959.56 u/sec=23.62	ppl=8.064299 
[2023-12-17:12:26:02:INFO:sockeye.training:__call__] E=22 B=89950	s/sec=2273.17 tok/sec=94951.04 u/sec=23.64	ppl=8.076224 
[2023-12-17:12:26:04:INFO:sockeye.training:__call__] E=22 B=90000	s/sec=1768.09 tok/sec=102079.46 u/sec=22.60	ppl=8.078272 
[2023-12-17:12:26:06:INFO:sockeye.training:__call__] E=22 B=90050	s/sec=2398.51 tok/sec=94021.44 u/sec=24.18	ppl=8.093981 
[2023-12-17:12:26:08:INFO:sockeye.training:__call__] E=22 B=90100	s/sec=2050.15 tok/sec=97438.80 u/sec=23.64	ppl=8.104998 
[2023-12-17:12:26:10:INFO:sockeye.training:__call__] E=22 B=90150	s/sec=2089.14 tok/sec=98783.91 u/sec=23.36	ppl=8.111726 
[2023-12-17:12:26:12:INFO:sockeye.training:__call__] E=22 B=90200	s/sec=2214.68 tok/sec=98383.18 u/sec=23.62	ppl=8.124278 
[2023-12-17:12:26:15:INFO:sockeye.training:__call__] E=22 B=90250	s/sec=2104.66 tok/sec=101960.91 u/sec=22.88	ppl=8.132759 
[2023-12-17:12:26:17:INFO:sockeye.training:__call__] E=22 B=90300	s/sec=2001.76 tok/sec=100562.66 u/sec=23.17	ppl=8.139170 
[2023-12-17:12:26:19:INFO:sockeye.training:__call__] E=22 B=90350	s/sec=1750.13 tok/sec=104223.36 u/sec=22.69	ppl=8.143618 
[2023-12-17:12:26:21:INFO:sockeye.training:__call__] E=22 B=90400	s/sec=2028.14 tok/sec=100894.41 u/sec=22.88	ppl=8.148310 
[2023-12-17:12:26:23:INFO:sockeye.training:__call__] E=22 B=90450	s/sec=1944.57 tok/sec=99656.28 u/sec=23.06	ppl=8.152051 
[2023-12-17:12:26:25:INFO:sockeye.training:__call__] E=22 B=90500	s/sec=2125.82 tok/sec=97824.37 u/sec=23.03	ppl=8.160944 
[2023-12-17:12:26:28:INFO:sockeye.training:__call__] E=22 B=90550	s/sec=2282.78 tok/sec=94324.30 u/sec=24.02	ppl=8.170304 
[2023-12-17:12:26:30:INFO:sockeye.training:__call__] E=22 B=90600	s/sec=1762.48 tok/sec=101703.22 u/sec=23.24	ppl=8.179457 
[2023-12-17:12:26:32:INFO:sockeye.training:__call__] E=22 B=90650	s/sec=2122.88 tok/sec=98453.40 u/sec=23.61	ppl=8.185735 
[2023-12-17:12:26:34:INFO:sockeye.training:__call__] E=22 B=90700	s/sec=2019.15 tok/sec=100443.33 u/sec=22.94	ppl=8.194206 
[2023-12-17:12:26:36:INFO:sockeye.training:__call__] E=22 B=90750	s/sec=2080.42 tok/sec=102854.40 u/sec=22.93	ppl=8.202190 
[2023-12-17:12:26:38:INFO:sockeye.training:__call__] E=22 B=90800	s/sec=2326.28 tok/sec=97065.19 u/sec=23.76	ppl=8.213530 
[2023-12-17:12:26:40:INFO:sockeye.training:__call__] E=22 B=90850	s/sec=1982.13 tok/sec=101040.30 u/sec=23.16	ppl=8.219073 
[2023-12-17:12:26:43:INFO:sockeye.training:__call__] E=22 B=90900	s/sec=2300.97 tok/sec=97749.66 u/sec=23.73	ppl=8.229367 
[2023-12-17:12:26:45:INFO:sockeye.training:__call__] E=22 B=90950	s/sec=1891.99 tok/sec=101483.98 u/sec=22.96	ppl=8.235350 
[2023-12-17:12:26:47:INFO:sockeye.training:__call__] E=22 B=91000	s/sec=2045.69 tok/sec=97983.08 u/sec=23.42	ppl=8.241601 
[2023-12-17:12:26:49:INFO:sockeye.training:__call__] E=22 B=91050	s/sec=2001.77 tok/sec=101091.23 u/sec=22.87	ppl=8.249103 
[2023-12-17:12:26:51:INFO:sockeye.training:__call__] E=22 B=91100	s/sec=2071.79 tok/sec=99178.84 u/sec=23.16	ppl=8.252113 
[2023-12-17:12:26:53:INFO:sockeye.training:__call__] E=22 B=91150	s/sec=2166.13 tok/sec=97835.08 u/sec=23.63	ppl=8.259324 
[2023-12-17:12:26:56:INFO:sockeye.training:__call__] E=22 B=91200	s/sec=1972.43 tok/sec=103598.11 u/sec=22.70	ppl=8.265624 
[2023-12-17:12:26:58:INFO:sockeye.training:__call__] E=22 B=91250	s/sec=2266.94 tok/sec=96337.53 u/sec=23.77	ppl=8.272195 
[2023-12-17:12:27:00:INFO:sockeye.training:__call__] E=22 B=91300	s/sec=2131.33 tok/sec=99110.64 u/sec=23.54	ppl=8.279450 
[2023-12-17:12:27:02:INFO:sockeye.training:__call__] E=22 B=91350	s/sec=1960.52 tok/sec=107943.98 u/sec=21.84	ppl=8.286165 
[2023-12-17:12:27:04:INFO:sockeye.training:__call__] E=22 B=91400	s/sec=1925.30 tok/sec=100916.55 u/sec=22.75	ppl=8.292078 
[2023-12-17:12:27:06:INFO:sockeye.training:__call__] E=22 B=91450	s/sec=2262.15 tok/sec=96223.48 u/sec=23.84	ppl=8.300497 
[2023-12-17:12:27:09:INFO:sockeye.training:__call__] E=22 B=91500	s/sec=2040.50 tok/sec=99991.85 u/sec=23.19	ppl=8.307018 
[2023-12-17:12:27:11:INFO:sockeye.training:__call__] E=22 B=91550	s/sec=2298.85 tok/sec=97238.63 u/sec=23.59	ppl=8.317038 
[2023-12-17:12:27:13:INFO:sockeye.training:__call__] E=22 B=91600	s/sec=2141.72 tok/sec=98458.83 u/sec=23.57	ppl=8.323484 
[2023-12-17:12:27:15:INFO:sockeye.training:__call__] E=22 B=91650	s/sec=2215.70 tok/sec=99811.99 u/sec=23.55	ppl=8.331444 
[2023-12-17:12:27:17:INFO:sockeye.training:__call__] E=22 B=91700	s/sec=2301.97 tok/sec=93096.68 u/sec=24.10	ppl=8.340017 
[2023-12-17:12:27:19:INFO:sockeye.training:__call__] E=22 B=91750	s/sec=2145.29 tok/sec=97894.50 u/sec=23.48	ppl=8.348316 
[2023-12-17:12:27:21:INFO:sockeye.training:__call__] E=22 B=91800	s/sec=2362.02 tok/sec=95174.97 u/sec=23.58	ppl=8.358826 
[2023-12-17:12:27:23:INFO:sockeye.training:__call__] E=22 B=91850	s/sec=2154.66 tok/sec=99219.49 u/sec=23.46	ppl=8.364689 
[2023-12-17:12:27:26:INFO:sockeye.training:__call__] E=23 B=91900	s/sec=1918.96 tok/sec=93450.09 u/sec=21.81	ppl=8.363632 
[2023-12-17:12:27:28:INFO:sockeye.training:__call__] E=23 B=91950	s/sec=2142.69 tok/sec=96637.35 u/sec=23.33	ppl=8.348428 
[2023-12-17:12:27:30:INFO:sockeye.training:__call__] E=23 B=92000	s/sec=2321.70 tok/sec=96352.26 u/sec=23.67	ppl=8.331606 
[2023-12-17:12:27:30:INFO:sockeye.training:_create_checkpoint] Checkpoint [23]	Updates=92000 Epoch=23 Samples=8199752 Time-cost=172.169 Updates/sec=23.233
[2023-12-17:12:27:30:INFO:sockeye.training:_create_checkpoint] Checkpoint [23]	Train-ppl=8.331606
[2023-12-17:12:27:33:INFO:sockeye.training:_evaluate] Checkpoint [23]	Validation-ppl=28.039788	Validation-bleu=0.180385	Validation-chrf=31.658328	Validation-rouge1=0.164596	Validation-rouge2=0.080206	Validation-rougel=0.157144	Validation-length-ratio-mse=1.245799	Validation-ter=94.841106	Validation-avg-sec-per-sent=0.003411	Validation-decode-walltime=1.705528
[2023-12-17:12:27:33:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 18 checkpoints, best so far: 31.983236
[2023-12-17:12:27:33:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 1.383042 > 0.000000 over 20 checkpoints
[2023-12-17:12:27:33:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [23]	Learning-rate=0.000162
[2023-12-17:12:27:33:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:27:33:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:27:33:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00023"
[2023-12-17:12:27:35:INFO:sockeye.training:__call__] E=23 B=92050	s/sec=881.95 tok/sec=37595.56 u/sec=8.92	ppl=7.386309 
[2023-12-17:12:27:38:INFO:sockeye.training:__call__] E=23 B=92100	s/sec=1924.83 tok/sec=99070.21 u/sec=22.78	ppl=7.214556 
[2023-12-17:12:27:40:INFO:sockeye.training:__call__] E=23 B=92150	s/sec=2042.54 tok/sec=100346.44 u/sec=23.38	ppl=7.233553 
[2023-12-17:12:27:42:INFO:sockeye.training:__call__] E=23 B=92200	s/sec=2067.43 tok/sec=99087.86 u/sec=23.28	ppl=7.270839 
[2023-12-17:12:27:44:INFO:sockeye.training:__call__] E=23 B=92250	s/sec=2183.30 tok/sec=95040.33 u/sec=23.90	ppl=7.307340 
[2023-12-17:12:27:46:INFO:sockeye.training:__call__] E=23 B=92300	s/sec=2140.69 tok/sec=95548.86 u/sec=23.85	ppl=7.346610 
[2023-12-17:12:27:48:INFO:sockeye.training:__call__] E=23 B=92350	s/sec=2157.69 tok/sec=100781.07 u/sec=23.17	ppl=7.381501 
[2023-12-17:12:27:50:INFO:sockeye.training:__call__] E=23 B=92400	s/sec=2187.89 tok/sec=98612.75 u/sec=23.50	ppl=7.390512 
[2023-12-17:12:27:53:INFO:sockeye.training:__call__] E=23 B=92450	s/sec=1779.73 tok/sec=103320.38 u/sec=23.03	ppl=7.381797 
[2023-12-17:12:27:55:INFO:sockeye.training:__call__] E=23 B=92500	s/sec=1950.14 tok/sec=100946.73 u/sec=23.22	ppl=7.376257 
[2023-12-17:12:27:57:INFO:sockeye.training:__call__] E=23 B=92550	s/sec=2148.85 tok/sec=100447.65 u/sec=23.16	ppl=7.387144 
[2023-12-17:12:27:59:INFO:sockeye.training:__call__] E=23 B=92600	s/sec=2005.06 tok/sec=99870.72 u/sec=23.42	ppl=7.403480 
[2023-12-17:12:28:01:INFO:sockeye.training:__call__] E=23 B=92650	s/sec=2103.58 tok/sec=100680.37 u/sec=22.79	ppl=7.414194 
[2023-12-17:12:28:03:INFO:sockeye.training:__call__] E=23 B=92700	s/sec=1908.30 tok/sec=103656.87 u/sec=22.38	ppl=7.410044 
[2023-12-17:12:28:06:INFO:sockeye.training:__call__] E=23 B=92750	s/sec=1836.52 tok/sec=102142.49 u/sec=22.86	ppl=7.415441 
[2023-12-17:12:28:08:INFO:sockeye.training:__call__] E=23 B=92800	s/sec=2089.70 tok/sec=97968.94 u/sec=23.41	ppl=7.422983 
[2023-12-17:12:28:10:INFO:sockeye.training:__call__] E=23 B=92850	s/sec=2051.21 tok/sec=99287.51 u/sec=23.14	ppl=7.440219 
[2023-12-17:12:28:12:INFO:sockeye.training:__call__] E=23 B=92900	s/sec=1667.56 tok/sec=104850.35 u/sec=22.17	ppl=7.450023 
[2023-12-17:12:28:14:INFO:sockeye.training:__call__] E=23 B=92950	s/sec=2111.63 tok/sec=98396.76 u/sec=23.61	ppl=7.462761 
[2023-12-17:12:28:16:INFO:sockeye.training:__call__] E=23 B=93000	s/sec=2339.54 tok/sec=95477.53 u/sec=23.89	ppl=7.479897 
[2023-12-17:12:28:19:INFO:sockeye.training:__call__] E=23 B=93050	s/sec=2251.31 tok/sec=99249.12 u/sec=23.03	ppl=7.492569 
[2023-12-17:12:28:21:INFO:sockeye.training:__call__] E=23 B=93100	s/sec=1801.72 tok/sec=103982.95 u/sec=22.75	ppl=7.497649 
[2023-12-17:12:28:23:INFO:sockeye.training:__call__] E=23 B=93150	s/sec=2121.46 tok/sec=98094.94 u/sec=23.34	ppl=7.511366 
[2023-12-17:12:28:25:INFO:sockeye.training:__call__] E=23 B=93200	s/sec=2038.07 tok/sec=99883.75 u/sec=22.95	ppl=7.522043 
[2023-12-17:12:28:27:INFO:sockeye.training:__call__] E=23 B=93250	s/sec=2047.22 tok/sec=97150.66 u/sec=23.56	ppl=7.528836 
[2023-12-17:12:28:29:INFO:sockeye.training:__call__] E=23 B=93300	s/sec=1907.19 tok/sec=100034.97 u/sec=23.60	ppl=7.538975 
[2023-12-17:12:28:32:INFO:sockeye.training:__call__] E=23 B=93350	s/sec=2167.57 tok/sec=98540.94 u/sec=23.24	ppl=7.552663 
[2023-12-17:12:28:34:INFO:sockeye.training:__call__] E=23 B=93400	s/sec=2213.41 tok/sec=95759.30 u/sec=23.81	ppl=7.563809 
[2023-12-17:12:28:36:INFO:sockeye.training:__call__] E=23 B=93450	s/sec=2236.20 tok/sec=97257.63 u/sec=23.65	ppl=7.575928 
[2023-12-17:12:28:38:INFO:sockeye.training:__call__] E=23 B=93500	s/sec=2116.35 tok/sec=99810.64 u/sec=23.49	ppl=7.585386 
[2023-12-17:12:28:40:INFO:sockeye.training:__call__] E=23 B=93550	s/sec=1994.03 tok/sec=100833.73 u/sec=23.12	ppl=7.593392 
[2023-12-17:12:28:42:INFO:sockeye.training:__call__] E=23 B=93600	s/sec=2077.26 tok/sec=99975.40 u/sec=23.18	ppl=7.603277 
[2023-12-17:12:28:44:INFO:sockeye.training:__call__] E=23 B=93650	s/sec=2069.30 tok/sec=97268.65 u/sec=23.64	ppl=7.614107 
[2023-12-17:12:28:46:INFO:sockeye.training:__call__] E=23 B=93700	s/sec=2063.36 tok/sec=100606.18 u/sec=23.07	ppl=7.624724 
[2023-12-17:12:28:49:INFO:sockeye.training:__call__] E=23 B=93750	s/sec=2086.40 tok/sec=99158.29 u/sec=23.41	ppl=7.640766 
[2023-12-17:12:28:51:INFO:sockeye.training:__call__] E=23 B=93800	s/sec=2121.88 tok/sec=99055.79 u/sec=23.23	ppl=7.653403 
[2023-12-17:12:28:53:INFO:sockeye.training:__call__] E=23 B=93850	s/sec=2072.00 tok/sec=98541.45 u/sec=23.04	ppl=7.663720 
[2023-12-17:12:28:55:INFO:sockeye.training:__call__] E=23 B=93900	s/sec=1753.24 tok/sec=102505.84 u/sec=22.83	ppl=7.665099 
[2023-12-17:12:28:57:INFO:sockeye.training:__call__] E=23 B=93950	s/sec=1989.31 tok/sec=99083.21 u/sec=23.41	ppl=7.675669 
[2023-12-17:12:28:59:INFO:sockeye.training:__call__] E=23 B=94000	s/sec=1784.38 tok/sec=102692.80 u/sec=22.76	ppl=7.675451 
[2023-12-17:12:29:02:INFO:sockeye.training:__call__] E=23 B=94050	s/sec=2199.25 tok/sec=98845.97 u/sec=23.54	ppl=7.685232 
[2023-12-17:12:29:04:INFO:sockeye.training:__call__] E=23 B=94100	s/sec=2083.57 tok/sec=100247.49 u/sec=23.05	ppl=7.697029 
[2023-12-17:12:29:06:INFO:sockeye.training:__call__] E=23 B=94150	s/sec=2028.77 tok/sec=100345.29 u/sec=23.39	ppl=7.708191 
[2023-12-17:12:29:08:INFO:sockeye.training:__call__] E=23 B=94200	s/sec=2077.64 tok/sec=100473.95 u/sec=23.35	ppl=7.718305 
[2023-12-17:12:29:10:INFO:sockeye.training:__call__] E=23 B=94250	s/sec=1997.16 tok/sec=101681.89 u/sec=23.07	ppl=7.727353 
[2023-12-17:12:29:12:INFO:sockeye.training:__call__] E=23 B=94300	s/sec=1903.98 tok/sec=101070.73 u/sec=23.20	ppl=7.734173 
[2023-12-17:12:29:15:INFO:sockeye.training:__call__] E=23 B=94350	s/sec=1838.49 tok/sec=101475.61 u/sec=23.12	ppl=7.739384 
[2023-12-17:12:29:17:INFO:sockeye.training:__call__] E=23 B=94400	s/sec=2239.14 tok/sec=96060.11 u/sec=23.60	ppl=7.747955 
[2023-12-17:12:29:19:INFO:sockeye.training:__call__] E=23 B=94450	s/sec=2235.14 tok/sec=98228.41 u/sec=22.98	ppl=7.756737 
[2023-12-17:12:29:21:INFO:sockeye.training:__call__] E=23 B=94500	s/sec=2205.25 tok/sec=98385.58 u/sec=23.52	ppl=7.765803 
[2023-12-17:12:29:23:INFO:sockeye.training:__call__] E=23 B=94550	s/sec=1970.13 tok/sec=100159.06 u/sec=23.36	ppl=7.771147 
[2023-12-17:12:29:25:INFO:sockeye.training:__call__] E=23 B=94600	s/sec=2055.17 tok/sec=101378.67 u/sec=23.19	ppl=7.781340 
[2023-12-17:12:29:27:INFO:sockeye.training:__call__] E=23 B=94650	s/sec=2104.33 tok/sec=101474.79 u/sec=22.79	ppl=7.789559 
[2023-12-17:12:29:29:INFO:sockeye.training:__call__] E=23 B=94700	s/sec=2359.10 tok/sec=94487.99 u/sec=24.21	ppl=7.800074 
[2023-12-17:12:29:32:INFO:sockeye.training:__call__] E=23 B=94750	s/sec=2222.16 tok/sec=99158.44 u/sec=22.99	ppl=7.809241 
[2023-12-17:12:29:34:INFO:sockeye.training:__call__] E=23 B=94800	s/sec=2310.82 tok/sec=94663.54 u/sec=23.87	ppl=7.819800 
[2023-12-17:12:29:36:INFO:sockeye.training:__call__] E=23 B=94850	s/sec=2183.14 tok/sec=96317.37 u/sec=23.81	ppl=7.828236 
[2023-12-17:12:29:38:INFO:sockeye.training:__call__] E=23 B=94900	s/sec=1961.99 tok/sec=101426.42 u/sec=22.75	ppl=7.833793 
[2023-12-17:12:29:40:INFO:sockeye.training:__call__] E=23 B=94950	s/sec=2258.82 tok/sec=98281.10 u/sec=23.37	ppl=7.841647 
[2023-12-17:12:29:42:INFO:sockeye.training:__call__] E=23 B=95000	s/sec=1933.49 tok/sec=101303.03 u/sec=22.89	ppl=7.849012 
[2023-12-17:12:29:44:INFO:sockeye.training:__call__] E=23 B=95050	s/sec=2085.80 tok/sec=96457.93 u/sec=23.83	ppl=7.857116 
[2023-12-17:12:29:47:INFO:sockeye.training:__call__] E=23 B=95100	s/sec=1996.19 tok/sec=102140.87 u/sec=22.77	ppl=7.861762 
[2023-12-17:12:29:49:INFO:sockeye.training:__call__] E=23 B=95150	s/sec=2030.93 tok/sec=100745.77 u/sec=22.75	ppl=7.869623 
[2023-12-17:12:29:51:INFO:sockeye.training:__call__] E=23 B=95200	s/sec=2065.01 tok/sec=98467.94 u/sec=23.17	ppl=7.876741 
[2023-12-17:12:29:53:INFO:sockeye.training:__call__] E=23 B=95250	s/sec=2252.06 tok/sec=95517.21 u/sec=23.46	ppl=7.886567 
[2023-12-17:12:29:55:INFO:sockeye.training:__call__] E=23 B=95300	s/sec=1995.69 tok/sec=101358.86 u/sec=22.89	ppl=7.893557 
[2023-12-17:12:29:58:INFO:sockeye.training:__call__] E=23 B=95350	s/sec=2002.22 tok/sec=105705.89 u/sec=22.19	ppl=7.897726 
[2023-12-17:12:30:00:INFO:sockeye.training:__call__] E=23 B=95400	s/sec=2173.20 tok/sec=96214.35 u/sec=23.79	ppl=7.905857 
[2023-12-17:12:30:02:INFO:sockeye.training:__call__] E=23 B=95450	s/sec=1925.11 tok/sec=99522.34 u/sec=22.79	ppl=7.913595 
[2023-12-17:12:30:04:INFO:sockeye.training:__call__] E=23 B=95500	s/sec=2160.78 tok/sec=96440.44 u/sec=23.20	ppl=7.922028 
[2023-12-17:12:30:06:INFO:sockeye.training:__call__] E=23 B=95550	s/sec=2107.28 tok/sec=98514.54 u/sec=23.39	ppl=7.928722 
[2023-12-17:12:30:08:INFO:sockeye.training:__call__] E=23 B=95600	s/sec=1974.03 tok/sec=99948.00 u/sec=23.32	ppl=7.937309 
[2023-12-17:12:30:11:INFO:sockeye.training:__call__] E=23 B=95650	s/sec=1882.49 tok/sec=102904.53 u/sec=22.85	ppl=7.944393 
[2023-12-17:12:30:13:INFO:sockeye.training:__call__] E=23 B=95700	s/sec=2141.42 tok/sec=98051.04 u/sec=23.28	ppl=7.953718 
[2023-12-17:12:30:15:INFO:sockeye.training:__call__] E=23 B=95750	s/sec=2010.67 tok/sec=100162.61 u/sec=23.19	ppl=7.960701 
[2023-12-17:12:30:17:INFO:sockeye.training:__call__] E=23 B=95800	s/sec=1967.86 tok/sec=101354.03 u/sec=22.73	ppl=7.967118 
[2023-12-17:12:30:19:INFO:sockeye.training:__call__] E=23 B=95850	s/sec=2378.55 tok/sec=96111.64 u/sec=23.67	ppl=7.976602 
[2023-12-17:12:30:21:INFO:sockeye.training:__call__] E=24 B=95900	s/sec=1882.10 tok/sec=96324.66 u/sec=21.54	ppl=7.976820 
[2023-12-17:12:30:24:INFO:sockeye.training:__call__] E=24 B=95950	s/sec=2168.27 tok/sec=95373.74 u/sec=23.45	ppl=7.964934 
[2023-12-17:12:30:26:INFO:sockeye.training:__call__] E=24 B=96000	s/sec=2131.10 tok/sec=99708.81 u/sec=23.20	ppl=7.952494 
[2023-12-17:12:30:26:INFO:sockeye.training:_create_checkpoint] Checkpoint [24]	Updates=96000 Epoch=24 Samples=8555896 Time-cost=172.428 Updates/sec=23.198
[2023-12-17:12:30:26:INFO:sockeye.training:_create_checkpoint] Checkpoint [24]	Train-ppl=7.952494
[2023-12-17:12:30:28:INFO:sockeye.training:_evaluate] Checkpoint [24]	Validation-ppl=29.021747	Validation-bleu=0.188692	Validation-chrf=32.780032	Validation-rouge1=0.172396	Validation-rouge2=0.087062	Validation-rougel=0.166430	Validation-length-ratio-mse=1.267278	Validation-ter=93.355345	Validation-avg-sec-per-sent=0.003228	Validation-decode-walltime=1.613806
[2023-12-17:12:30:28:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 32.780032 (delta=0.796795).
[2023-12-17:12:30:28:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 1.861041 > 0.000000 over 20 checkpoints
[2023-12-17:12:30:28:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [24]	Learning-rate=0.000162
[2023-12-17:12:30:29:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:30:29:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:30:29:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00024"
[2023-12-17:12:30:29:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00024'
[2023-12-17:12:30:30:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:12:30:30:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl'
[2023-12-17:12:30:32:INFO:sockeye.training:__call__] E=24 B=96050	s/sec=685.01 tok/sec=36158.10 u/sec=8.08	ppl=7.045107 
[2023-12-17:12:30:34:INFO:sockeye.training:__call__] E=24 B=96100	s/sec=2164.97 tok/sec=98859.70 u/sec=23.25	ppl=7.097912 
[2023-12-17:12:30:36:INFO:sockeye.training:__call__] E=24 B=96150	s/sec=1974.35 tok/sec=97451.28 u/sec=23.55	ppl=7.050912 
[2023-12-17:12:30:38:INFO:sockeye.training:__call__] E=24 B=96200	s/sec=2116.39 tok/sec=100521.09 u/sec=23.12	ppl=7.070229 
[2023-12-17:12:30:41:INFO:sockeye.training:__call__] E=24 B=96250	s/sec=2214.18 tok/sec=95821.77 u/sec=23.62	ppl=7.088746 
[2023-12-17:12:30:43:INFO:sockeye.training:__call__] E=24 B=96300	s/sec=1867.32 tok/sec=102887.28 u/sec=22.66	ppl=7.069587 
[2023-12-17:12:30:45:INFO:sockeye.training:__call__] E=24 B=96350	s/sec=2116.12 tok/sec=99277.34 u/sec=23.00	ppl=7.078230 
[2023-12-17:12:30:47:INFO:sockeye.training:__call__] E=24 B=96400	s/sec=2113.74 tok/sec=101253.89 u/sec=22.94	ppl=7.094035 
[2023-12-17:12:30:49:INFO:sockeye.training:__call__] E=24 B=96450	s/sec=1703.95 tok/sec=103013.37 u/sec=22.47	ppl=7.081947 
[2023-12-17:12:30:51:INFO:sockeye.training:__call__] E=24 B=96500	s/sec=2009.13 tok/sec=101737.24 u/sec=22.87	ppl=7.106112 
[2023-12-17:12:30:54:INFO:sockeye.training:__call__] E=24 B=96550	s/sec=2136.47 tok/sec=100886.16 u/sec=23.22	ppl=7.116306 
[2023-12-17:12:30:56:INFO:sockeye.training:__call__] E=24 B=96600	s/sec=2054.94 tok/sec=100251.24 u/sec=23.35	ppl=7.127340 
[2023-12-17:12:30:58:INFO:sockeye.training:__call__] E=24 B=96650	s/sec=1990.06 tok/sec=100050.63 u/sec=22.82	ppl=7.138362 
[2023-12-17:12:31:00:INFO:sockeye.training:__call__] E=24 B=96700	s/sec=2140.52 tok/sec=100881.62 u/sec=23.11	ppl=7.156782 
[2023-12-17:12:31:02:INFO:sockeye.training:__call__] E=24 B=96750	s/sec=2199.06 tok/sec=96983.56 u/sec=23.41	ppl=7.162682 
[2023-12-17:12:31:04:INFO:sockeye.training:__call__] E=24 B=96800	s/sec=2180.87 tok/sec=97860.92 u/sec=22.87	ppl=7.182753 
[2023-12-17:12:31:07:INFO:sockeye.training:__call__] E=24 B=96850	s/sec=2047.26 tok/sec=100944.40 u/sec=22.97	ppl=7.188651 
[2023-12-17:12:31:09:INFO:sockeye.training:__call__] E=24 B=96900	s/sec=1673.97 tok/sec=104369.10 u/sec=22.50	ppl=7.187093 
[2023-12-17:12:31:11:INFO:sockeye.training:__call__] E=24 B=96950	s/sec=2057.23 tok/sec=100398.87 u/sec=23.04	ppl=7.189487 
[2023-12-17:12:31:13:INFO:sockeye.training:__call__] E=24 B=97000	s/sec=1995.86 tok/sec=101438.18 u/sec=23.06	ppl=7.198443 
[2023-12-17:12:31:15:INFO:sockeye.training:__call__] E=24 B=97050	s/sec=2301.57 tok/sec=96181.33 u/sec=23.66	ppl=7.212840 
[2023-12-17:12:31:17:INFO:sockeye.training:__call__] E=24 B=97100	s/sec=1688.85 tok/sec=103197.87 u/sec=23.15	ppl=7.214406 
[2023-12-17:12:31:20:INFO:sockeye.training:__call__] E=24 B=97150	s/sec=2158.33 tok/sec=96901.48 u/sec=23.26	ppl=7.230911 
[2023-12-17:12:31:22:INFO:sockeye.training:__call__] E=24 B=97200	s/sec=1989.40 tok/sec=101738.49 u/sec=23.24	ppl=7.241206 
[2023-12-17:12:31:24:INFO:sockeye.training:__call__] E=24 B=97250	s/sec=2108.81 tok/sec=96674.08 u/sec=23.54	ppl=7.252981 
[2023-12-17:12:31:26:INFO:sockeye.training:__call__] E=24 B=97300	s/sec=2086.09 tok/sec=99714.24 u/sec=23.32	ppl=7.261254 
[2023-12-17:12:31:28:INFO:sockeye.training:__call__] E=24 B=97350	s/sec=2062.08 tok/sec=98009.22 u/sec=23.69	ppl=7.267883 
[2023-12-17:12:31:30:INFO:sockeye.training:__call__] E=24 B=97400	s/sec=2194.34 tok/sec=98511.43 u/sec=22.86	ppl=7.281523 
[2023-12-17:12:31:33:INFO:sockeye.training:__call__] E=24 B=97450	s/sec=2091.14 tok/sec=100666.30 u/sec=22.77	ppl=7.290224 
[2023-12-17:12:31:35:INFO:sockeye.training:__call__] E=24 B=97500	s/sec=2274.23 tok/sec=97901.65 u/sec=23.65	ppl=7.304345 
[2023-12-17:12:31:37:INFO:sockeye.training:__call__] E=24 B=97550	s/sec=2032.04 tok/sec=103256.48 u/sec=22.68	ppl=7.315236 
[2023-12-17:12:31:39:INFO:sockeye.training:__call__] E=24 B=97600	s/sec=2055.15 tok/sec=102046.96 u/sec=22.90	ppl=7.324075 
[2023-12-17:12:31:41:INFO:sockeye.training:__call__] E=24 B=97650	s/sec=2179.11 tok/sec=97558.16 u/sec=23.40	ppl=7.334961 
[2023-12-17:12:31:43:INFO:sockeye.training:__call__] E=24 B=97700	s/sec=2002.16 tok/sec=100810.93 u/sec=23.13	ppl=7.341292 
[2023-12-17:12:31:45:INFO:sockeye.training:__call__] E=24 B=97750	s/sec=2213.30 tok/sec=97369.98 u/sec=23.65	ppl=7.351472 
[2023-12-17:12:31:48:INFO:sockeye.training:__call__] E=24 B=97800	s/sec=2164.16 tok/sec=99135.71 u/sec=23.61	ppl=7.362552 
[2023-12-17:12:31:50:INFO:sockeye.training:__call__] E=24 B=97850	s/sec=2163.42 tok/sec=96982.32 u/sec=23.93	ppl=7.372963 
[2023-12-17:12:31:52:INFO:sockeye.training:__call__] E=24 B=97900	s/sec=1986.61 tok/sec=99359.74 u/sec=22.82	ppl=7.380966 
[2023-12-17:12:31:54:INFO:sockeye.training:__call__] E=24 B=97950	s/sec=1934.90 tok/sec=98849.14 u/sec=22.99	ppl=7.385508 
[2023-12-17:12:31:56:INFO:sockeye.training:__call__] E=24 B=98000	s/sec=2031.33 tok/sec=100314.79 u/sec=22.88	ppl=7.394004 
[2023-12-17:12:31:58:INFO:sockeye.training:__call__] E=24 B=98050	s/sec=2106.34 tok/sec=96606.82 u/sec=23.42	ppl=7.404017 
[2023-12-17:12:32:01:INFO:sockeye.training:__call__] E=24 B=98100	s/sec=1811.05 tok/sec=101154.10 u/sec=22.96	ppl=7.407704 
[2023-12-17:12:32:03:INFO:sockeye.training:__call__] E=24 B=98150	s/sec=1910.84 tok/sec=102330.42 u/sec=22.84	ppl=7.414746 
[2023-12-17:12:32:05:INFO:sockeye.training:__call__] E=24 B=98200	s/sec=2284.30 tok/sec=95718.78 u/sec=23.91	ppl=7.426424 
[2023-12-17:12:32:07:INFO:sockeye.training:__call__] E=24 B=98250	s/sec=1992.23 tok/sec=107220.67 u/sec=22.04	ppl=7.433461 
[2023-12-17:12:32:09:INFO:sockeye.training:__call__] E=24 B=98300	s/sec=2255.08 tok/sec=96611.68 u/sec=23.73	ppl=7.443245 
[2023-12-17:12:32:11:INFO:sockeye.training:__call__] E=24 B=98350	s/sec=2031.53 tok/sec=98156.38 u/sec=23.91	ppl=7.451296 
[2023-12-17:12:32:13:INFO:sockeye.training:__call__] E=24 B=98400	s/sec=2088.86 tok/sec=100028.85 u/sec=23.11	ppl=7.463285 
[2023-12-17:12:32:16:INFO:sockeye.training:__call__] E=24 B=98450	s/sec=2259.20 tok/sec=97918.94 u/sec=23.69	ppl=7.472140 
[2023-12-17:12:32:18:INFO:sockeye.training:__call__] E=24 B=98500	s/sec=2056.20 tok/sec=96407.91 u/sec=23.54	ppl=7.480609 
[2023-12-17:12:32:20:INFO:sockeye.training:__call__] E=24 B=98550	s/sec=1895.12 tok/sec=101624.94 u/sec=22.69	ppl=7.485402 
[2023-12-17:12:32:22:INFO:sockeye.training:__call__] E=24 B=98600	s/sec=2096.67 tok/sec=100341.94 u/sec=23.32	ppl=7.492966 
[2023-12-17:12:32:24:INFO:sockeye.training:__call__] E=24 B=98650	s/sec=2000.07 tok/sec=104625.84 u/sec=22.60	ppl=7.498083 
[2023-12-17:12:32:26:INFO:sockeye.training:__call__] E=24 B=98700	s/sec=2091.64 tok/sec=98145.42 u/sec=23.47	ppl=7.505832 
[2023-12-17:12:32:28:INFO:sockeye.training:__call__] E=24 B=98750	s/sec=2204.79 tok/sec=97703.22 u/sec=23.52	ppl=7.516422 
[2023-12-17:12:32:31:INFO:sockeye.training:__call__] E=24 B=98800	s/sec=2231.78 tok/sec=96296.50 u/sec=23.97	ppl=7.525286 
[2023-12-17:12:32:33:INFO:sockeye.training:__call__] E=24 B=98850	s/sec=2113.14 tok/sec=95350.99 u/sec=24.23	ppl=7.532347 
[2023-12-17:12:32:35:INFO:sockeye.training:__call__] E=24 B=98900	s/sec=2168.01 tok/sec=98513.24 u/sec=23.44	ppl=7.540274 
[2023-12-17:12:32:37:INFO:sockeye.training:__call__] E=24 B=98950	s/sec=2061.73 tok/sec=96325.43 u/sec=23.69	ppl=7.545046 
[2023-12-17:12:32:39:INFO:sockeye.training:__call__] E=24 B=99000	s/sec=1999.83 tok/sec=101412.22 u/sec=22.77	ppl=7.550136 
[2023-12-17:12:32:41:INFO:sockeye.training:__call__] E=24 B=99050	s/sec=2245.91 tok/sec=95825.45 u/sec=23.75	ppl=7.559895 
[2023-12-17:12:32:43:INFO:sockeye.training:__call__] E=24 B=99100	s/sec=2038.00 tok/sec=97400.36 u/sec=23.63	ppl=7.564138 
[2023-12-17:12:32:45:INFO:sockeye.training:__call__] E=24 B=99150	s/sec=2101.72 tok/sec=99877.96 u/sec=23.09	ppl=7.570593 
[2023-12-17:12:32:48:INFO:sockeye.training:__call__] E=24 B=99200	s/sec=1904.85 tok/sec=101324.41 u/sec=23.21	ppl=7.574962 
[2023-12-17:12:32:50:INFO:sockeye.training:__call__] E=24 B=99250	s/sec=2136.91 tok/sec=97584.09 u/sec=23.47	ppl=7.582547 
[2023-12-17:12:32:52:INFO:sockeye.training:__call__] E=24 B=99300	s/sec=2044.57 tok/sec=99062.84 u/sec=23.28	ppl=7.589976 
[2023-12-17:12:32:54:INFO:sockeye.training:__call__] E=24 B=99350	s/sec=1926.51 tok/sec=101465.35 u/sec=23.11	ppl=7.597073 
[2023-12-17:12:32:56:INFO:sockeye.training:__call__] E=24 B=99400	s/sec=1861.06 tok/sec=104598.15 u/sec=22.76	ppl=7.602206 
[2023-12-17:12:32:58:INFO:sockeye.training:__call__] E=24 B=99450	s/sec=2118.37 tok/sec=99726.20 u/sec=23.15	ppl=7.608583 
[2023-12-17:12:33:01:INFO:sockeye.training:__call__] E=24 B=99500	s/sec=2379.80 tok/sec=95404.67 u/sec=23.72	ppl=7.617583 
[2023-12-17:12:33:03:INFO:sockeye.training:__call__] E=24 B=99550	s/sec=2056.55 tok/sec=99847.14 u/sec=23.29	ppl=7.623657 
[2023-12-17:12:33:05:INFO:sockeye.training:__call__] E=24 B=99600	s/sec=2021.59 tok/sec=96945.77 u/sec=23.53	ppl=7.629213 
[2023-12-17:12:33:07:INFO:sockeye.training:__call__] E=24 B=99650	s/sec=1914.47 tok/sec=100070.96 u/sec=23.14	ppl=7.635230 
[2023-12-17:12:33:09:INFO:sockeye.training:__call__] E=24 B=99700	s/sec=2291.38 tok/sec=97694.93 u/sec=23.59	ppl=7.641862 
[2023-12-17:12:33:11:INFO:sockeye.training:__call__] E=24 B=99750	s/sec=2186.01 tok/sec=98264.96 u/sec=23.56	ppl=7.650465 
[2023-12-17:12:33:13:INFO:sockeye.training:__call__] E=24 B=99800	s/sec=2142.76 tok/sec=99245.14 u/sec=23.29	ppl=7.657914 
[2023-12-17:12:33:16:INFO:sockeye.training:__call__] E=24 B=99850	s/sec=2096.54 tok/sec=97315.40 u/sec=23.36	ppl=7.663302 
[2023-12-17:12:33:18:INFO:sockeye.training:__call__] E=25 B=99900	s/sec=1936.49 tok/sec=93439.03 u/sec=22.41	ppl=7.660682 
[2023-12-17:12:33:20:INFO:sockeye.training:__call__] E=25 B=99950	s/sec=1956.62 tok/sec=99435.21 u/sec=23.43	ppl=7.646830 
[2023-12-17:12:33:22:INFO:sockeye.training:__call__] E=25 B=100000	s/sec=1920.78 tok/sec=102899.23 u/sec=22.36	ppl=7.635172 
[2023-12-17:12:33:22:INFO:sockeye.training:_create_checkpoint] Checkpoint [25]	Updates=100000 Epoch=25 Samples=8911768 Time-cost=172.360 Updates/sec=23.207
[2023-12-17:12:33:22:INFO:sockeye.training:_create_checkpoint] Checkpoint [25]	Train-ppl=7.635172
[2023-12-17:12:33:25:INFO:sockeye.training:_evaluate] Checkpoint [25]	Validation-ppl=29.364699	Validation-bleu=0.157933	Validation-chrf=30.289782	Validation-rouge1=0.154094	Validation-rouge2=0.070716	Validation-rougel=0.146999	Validation-length-ratio-mse=1.275279	Validation-ter=96.244325	Validation-avg-sec-per-sent=0.003349	Validation-decode-walltime=1.674365
[2023-12-17:12:33:25:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 1 checkpoints, best so far: 32.780032
[2023-12-17:12:33:25:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:33:25:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [25]	Learning-rate=0.000162
[2023-12-17:12:33:25:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:33:25:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:33:26:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00025"
[2023-12-17:12:33:28:INFO:sockeye.training:__call__] E=25 B=100050	s/sec=776.43 tok/sec=36827.15 u/sec=8.62	ppl=6.780704 
[2023-12-17:12:33:30:INFO:sockeye.training:__call__] E=25 B=100100	s/sec=1937.84 tok/sec=98691.50 u/sec=23.34	ppl=6.691622 
[2023-12-17:12:33:32:INFO:sockeye.training:__call__] E=25 B=100150	s/sec=1993.64 tok/sec=97813.01 u/sec=23.64	ppl=6.732342 
[2023-12-17:12:33:34:INFO:sockeye.training:__call__] E=25 B=100200	s/sec=2118.25 tok/sec=97939.92 u/sec=23.35	ppl=6.759120 
[2023-12-17:12:33:36:INFO:sockeye.training:__call__] E=25 B=100250	s/sec=2361.16 tok/sec=96144.30 u/sec=23.69	ppl=6.790843 
[2023-12-17:12:33:39:INFO:sockeye.training:__call__] E=25 B=100300	s/sec=2125.60 tok/sec=98533.89 u/sec=22.94	ppl=6.817435 
[2023-12-17:12:33:41:INFO:sockeye.training:__call__] E=25 B=100350	s/sec=1923.53 tok/sec=100683.08 u/sec=22.90	ppl=6.809359 
[2023-12-17:12:33:43:INFO:sockeye.training:__call__] E=25 B=100400	s/sec=2094.60 tok/sec=99194.03 u/sec=23.38	ppl=6.825815 
[2023-12-17:12:33:45:INFO:sockeye.training:__call__] E=25 B=100450	s/sec=2078.46 tok/sec=99912.39 u/sec=22.87	ppl=6.831058 
[2023-12-17:12:33:47:INFO:sockeye.training:__call__] E=25 B=100500	s/sec=2407.44 tok/sec=94889.01 u/sec=23.92	ppl=6.873054 
[2023-12-17:12:33:49:INFO:sockeye.training:__call__] E=25 B=100550	s/sec=2280.53 tok/sec=97586.91 u/sec=23.80	ppl=6.889689 
[2023-12-17:12:33:51:INFO:sockeye.training:__call__] E=25 B=100600	s/sec=1915.39 tok/sec=102564.96 u/sec=22.93	ppl=6.901435 
[2023-12-17:12:33:54:INFO:sockeye.training:__call__] E=25 B=100650	s/sec=2096.84 tok/sec=99701.79 u/sec=23.11	ppl=6.911178 
[2023-12-17:12:33:56:INFO:sockeye.training:__call__] E=25 B=100700	s/sec=2173.44 tok/sec=99042.29 u/sec=22.68	ppl=6.926534 
[2023-12-17:12:33:58:INFO:sockeye.training:__call__] E=25 B=100750	s/sec=2070.73 tok/sec=98709.68 u/sec=23.28	ppl=6.938742 
[2023-12-17:12:34:00:INFO:sockeye.training:__call__] E=25 B=100800	s/sec=1901.62 tok/sec=102512.19 u/sec=22.86	ppl=6.945803 
[2023-12-17:12:34:02:INFO:sockeye.training:__call__] E=25 B=100850	s/sec=1812.99 tok/sec=103607.79 u/sec=22.57	ppl=6.947354 
[2023-12-17:12:34:05:INFO:sockeye.training:__call__] E=25 B=100900	s/sec=1967.45 tok/sec=99733.89 u/sec=23.24	ppl=6.956008 
[2023-12-17:12:34:07:INFO:sockeye.training:__call__] E=25 B=100950	s/sec=2071.43 tok/sec=97248.92 u/sec=23.33	ppl=6.967165 
[2023-12-17:12:34:09:INFO:sockeye.training:__call__] E=25 B=101000	s/sec=2125.40 tok/sec=97670.45 u/sec=23.59	ppl=6.977736 
[2023-12-17:12:34:11:INFO:sockeye.training:__call__] E=25 B=101050	s/sec=2196.24 tok/sec=100734.20 u/sec=23.46	ppl=6.982303 
[2023-12-17:12:34:13:INFO:sockeye.training:__call__] E=25 B=101100	s/sec=2170.01 tok/sec=99170.70 u/sec=23.34	ppl=6.996416 
[2023-12-17:12:34:15:INFO:sockeye.training:__call__] E=25 B=101150	s/sec=2006.88 tok/sec=100134.55 u/sec=23.40	ppl=7.004556 
[2023-12-17:12:34:17:INFO:sockeye.training:__call__] E=25 B=101200	s/sec=2166.19 tok/sec=95632.36 u/sec=23.79	ppl=7.021227 
[2023-12-17:12:34:19:INFO:sockeye.training:__call__] E=25 B=101250	s/sec=2056.37 tok/sec=101034.61 u/sec=23.33	ppl=7.031456 
[2023-12-17:12:34:22:INFO:sockeye.training:__call__] E=25 B=101300	s/sec=1927.07 tok/sec=102707.76 u/sec=23.12	ppl=7.031677 
[2023-12-17:12:34:24:INFO:sockeye.training:__call__] E=25 B=101350	s/sec=1992.83 tok/sec=101232.60 u/sec=22.81	ppl=7.035524 
[2023-12-17:12:34:26:INFO:sockeye.training:__call__] E=25 B=101400	s/sec=1857.34 tok/sec=109046.53 u/sec=21.50	ppl=7.039307 
[2023-12-17:12:34:28:INFO:sockeye.training:__call__] E=25 B=101450	s/sec=2012.50 tok/sec=101283.83 u/sec=22.87	ppl=7.049417 
[2023-12-17:12:34:30:INFO:sockeye.training:__call__] E=25 B=101500	s/sec=2177.88 tok/sec=96662.14 u/sec=23.31	ppl=7.061490 
[2023-12-17:12:34:33:INFO:sockeye.training:__call__] E=25 B=101550	s/sec=2268.68 tok/sec=98097.19 u/sec=23.44	ppl=7.075727 
[2023-12-17:12:34:35:INFO:sockeye.training:__call__] E=25 B=101600	s/sec=1975.43 tok/sec=102060.38 u/sec=22.99	ppl=7.083187 
[2023-12-17:12:34:37:INFO:sockeye.training:__call__] E=25 B=101650	s/sec=2268.67 tok/sec=96877.72 u/sec=23.83	ppl=7.095295 
[2023-12-17:12:34:39:INFO:sockeye.training:__call__] E=25 B=101700	s/sec=2080.85 tok/sec=100424.82 u/sec=23.60	ppl=7.102428 
[2023-12-17:12:34:41:INFO:sockeye.training:__call__] E=25 B=101750	s/sec=2142.99 tok/sec=99637.66 u/sec=23.33	ppl=7.112371 
[2023-12-17:12:34:43:INFO:sockeye.training:__call__] E=25 B=101800	s/sec=2228.08 tok/sec=97173.07 u/sec=23.64	ppl=7.121199 
[2023-12-17:12:34:45:INFO:sockeye.training:__call__] E=25 B=101850	s/sec=2170.14 tok/sec=96573.28 u/sec=23.26	ppl=7.131660 
[2023-12-17:12:34:48:INFO:sockeye.training:__call__] E=25 B=101900	s/sec=2034.04 tok/sec=100882.68 u/sec=23.07	ppl=7.139734 
[2023-12-17:12:34:50:INFO:sockeye.training:__call__] E=25 B=101950	s/sec=2105.88 tok/sec=97906.43 u/sec=23.63	ppl=7.145896 
[2023-12-17:12:34:52:INFO:sockeye.training:__call__] E=25 B=102000	s/sec=2007.70 tok/sec=95887.29 u/sec=23.54	ppl=7.154744 
[2023-12-17:12:34:54:INFO:sockeye.training:__call__] E=25 B=102050	s/sec=2148.75 tok/sec=99316.48 u/sec=23.15	ppl=7.166014 
[2023-12-17:12:34:56:INFO:sockeye.training:__call__] E=25 B=102100	s/sec=2026.84 tok/sec=99846.83 u/sec=23.24	ppl=7.171026 
[2023-12-17:12:34:58:INFO:sockeye.training:__call__] E=25 B=102150	s/sec=2023.34 tok/sec=97911.46 u/sec=23.77	ppl=7.178161 
[2023-12-17:12:35:00:INFO:sockeye.training:__call__] E=25 B=102200	s/sec=2107.29 tok/sec=98818.66 u/sec=23.35	ppl=7.183614 
[2023-12-17:12:35:02:INFO:sockeye.training:__call__] E=25 B=102250	s/sec=2104.44 tok/sec=96621.17 u/sec=23.83	ppl=7.191474 
[2023-12-17:12:35:05:INFO:sockeye.training:__call__] E=25 B=102300	s/sec=1791.75 tok/sec=104718.06 u/sec=22.22	ppl=7.195700 
[2023-12-17:12:35:07:INFO:sockeye.training:__call__] E=25 B=102350	s/sec=2137.95 tok/sec=99682.77 u/sec=22.96	ppl=7.205597 
[2023-12-17:12:35:09:INFO:sockeye.training:__call__] E=25 B=102400	s/sec=2165.95 tok/sec=97291.77 u/sec=23.92	ppl=7.211949 
[2023-12-17:12:35:11:INFO:sockeye.training:__call__] E=25 B=102450	s/sec=2101.90 tok/sec=99437.88 u/sec=23.17	ppl=7.220029 
[2023-12-17:12:35:13:INFO:sockeye.training:__call__] E=25 B=102500	s/sec=2083.74 tok/sec=97902.04 u/sec=23.30	ppl=7.227154 
[2023-12-17:12:35:15:INFO:sockeye.training:__call__] E=25 B=102550	s/sec=2161.03 tok/sec=95633.07 u/sec=23.78	ppl=7.234617 
[2023-12-17:12:35:17:INFO:sockeye.training:__call__] E=25 B=102600	s/sec=2329.53 tok/sec=93517.70 u/sec=23.91	ppl=7.241856 
[2023-12-17:12:35:20:INFO:sockeye.training:__call__] E=25 B=102650	s/sec=1751.11 tok/sec=104607.30 u/sec=22.43	ppl=7.243086 
[2023-12-17:12:35:22:INFO:sockeye.training:__call__] E=25 B=102700	s/sec=2138.55 tok/sec=99069.07 u/sec=23.12	ppl=7.252809 
[2023-12-17:12:35:24:INFO:sockeye.training:__call__] E=25 B=102750	s/sec=2058.14 tok/sec=101632.66 u/sec=22.89	ppl=7.257495 
[2023-12-17:12:35:26:INFO:sockeye.training:__call__] E=25 B=102800	s/sec=2431.19 tok/sec=94954.12 u/sec=23.89	ppl=7.267142 
[2023-12-17:12:35:28:INFO:sockeye.training:__call__] E=25 B=102850	s/sec=2084.97 tok/sec=98818.68 u/sec=23.44	ppl=7.272386 
[2023-12-17:12:35:30:INFO:sockeye.training:__call__] E=25 B=102900	s/sec=2097.90 tok/sec=97276.72 u/sec=23.46	ppl=7.279359 
[2023-12-17:12:35:33:INFO:sockeye.training:__call__] E=25 B=102950	s/sec=1805.07 tok/sec=100814.66 u/sec=23.36	ppl=7.284813 
[2023-12-17:12:35:35:INFO:sockeye.training:__call__] E=25 B=103000	s/sec=1785.40 tok/sec=101165.49 u/sec=23.10	ppl=7.289617 
[2023-12-17:12:35:37:INFO:sockeye.training:__call__] E=25 B=103050	s/sec=2121.03 tok/sec=97064.27 u/sec=23.46	ppl=7.295165 
[2023-12-17:12:35:39:INFO:sockeye.training:__call__] E=25 B=103100	s/sec=2085.21 tok/sec=99089.78 u/sec=22.99	ppl=7.302231 
[2023-12-17:12:35:41:INFO:sockeye.training:__call__] E=25 B=103150	s/sec=2114.53 tok/sec=100408.81 u/sec=22.98	ppl=7.309158 
[2023-12-17:12:35:43:INFO:sockeye.training:__call__] E=25 B=103200	s/sec=1899.63 tok/sec=104014.40 u/sec=22.53	ppl=7.315455 
[2023-12-17:12:35:46:INFO:sockeye.training:__call__] E=25 B=103250	s/sec=1949.05 tok/sec=102556.79 u/sec=22.98	ppl=7.322729 
[2023-12-17:12:35:48:INFO:sockeye.training:__call__] E=25 B=103300	s/sec=1862.09 tok/sec=100995.97 u/sec=23.09	ppl=7.329313 
[2023-12-17:12:35:50:INFO:sockeye.training:__call__] E=25 B=103350	s/sec=2143.63 tok/sec=98818.25 u/sec=23.59	ppl=7.337860 
[2023-12-17:12:35:52:INFO:sockeye.training:__call__] E=25 B=103400	s/sec=2310.26 tok/sec=98182.34 u/sec=23.36	ppl=7.345265 
[2023-12-17:12:35:54:INFO:sockeye.training:__call__] E=25 B=103450	s/sec=2055.33 tok/sec=97444.73 u/sec=23.66	ppl=7.352246 
[2023-12-17:12:35:56:INFO:sockeye.training:__call__] E=25 B=103500	s/sec=2015.63 tok/sec=102115.70 u/sec=22.90	ppl=7.357813 
[2023-12-17:12:35:58:INFO:sockeye.training:__call__] E=25 B=103550	s/sec=2198.68 tok/sec=99763.92 u/sec=23.49	ppl=7.365727 
[2023-12-17:12:36:01:INFO:sockeye.training:__call__] E=25 B=103600	s/sec=2050.42 tok/sec=100385.15 u/sec=23.26	ppl=7.373078 
[2023-12-17:12:36:03:INFO:sockeye.training:__call__] E=25 B=103650	s/sec=1992.18 tok/sec=100809.98 u/sec=22.89	ppl=7.378618 
[2023-12-17:12:36:05:INFO:sockeye.training:__call__] E=25 B=103700	s/sec=2059.41 tok/sec=96691.02 u/sec=23.45	ppl=7.382790 
[2023-12-17:12:36:07:INFO:sockeye.training:__call__] E=25 B=103750	s/sec=2259.47 tok/sec=95629.64 u/sec=23.58	ppl=7.391073 
[2023-12-17:12:36:09:INFO:sockeye.training:__call__] E=25 B=103800	s/sec=1991.74 tok/sec=100973.83 u/sec=23.05	ppl=7.396210 
[2023-12-17:12:36:11:INFO:sockeye.training:__call__] E=25 B=103850	s/sec=1956.16 tok/sec=101059.50 u/sec=22.94	ppl=7.402247 
[2023-12-17:12:36:14:INFO:sockeye.training:__call__] E=26 B=103900	s/sec=1899.20 tok/sec=95478.17 u/sec=22.19	ppl=7.398830 
[2023-12-17:12:36:16:INFO:sockeye.training:__call__] E=26 B=103950	s/sec=1965.49 tok/sec=101167.23 u/sec=23.18	ppl=7.385925 
[2023-12-17:12:36:18:INFO:sockeye.training:__call__] E=26 B=104000	s/sec=2142.07 tok/sec=100214.96 u/sec=23.12	ppl=7.373765 
[2023-12-17:12:36:18:INFO:sockeye.training:_create_checkpoint] Checkpoint [26]	Updates=104000 Epoch=26 Samples=9268368 Time-cost=172.240 Updates/sec=23.223
[2023-12-17:12:36:18:INFO:sockeye.training:_create_checkpoint] Checkpoint [26]	Train-ppl=7.373765
[2023-12-17:12:36:21:INFO:sockeye.training:_evaluate] Checkpoint [26]	Validation-ppl=29.366776	Validation-bleu=0.169713	Validation-chrf=32.052151	Validation-rouge1=0.169844	Validation-rouge2=0.086825	Validation-rougel=0.163671	Validation-length-ratio-mse=1.338613	Validation-ter=97.936442	Validation-avg-sec-per-sent=0.003355	Validation-decode-walltime=1.677655
[2023-12-17:12:36:21:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 2 checkpoints, best so far: 32.780032
[2023-12-17:12:36:21:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:36:21:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [26]	Learning-rate=0.000162
[2023-12-17:12:36:21:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:36:21:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:36:22:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00026"
[2023-12-17:12:36:24:INFO:sockeye.training:__call__] E=26 B=104050	s/sec=698.13 tok/sec=38182.81 u/sec=8.49	ppl=6.509015 
[2023-12-17:12:36:26:INFO:sockeye.training:__call__] E=26 B=104100	s/sec=1994.06 tok/sec=98660.30 u/sec=22.78	ppl=6.529709 
[2023-12-17:12:36:28:INFO:sockeye.training:__call__] E=26 B=104150	s/sec=2146.83 tok/sec=97660.52 u/sec=23.25	ppl=6.578500 
[2023-12-17:12:36:30:INFO:sockeye.training:__call__] E=26 B=104200	s/sec=2142.34 tok/sec=99794.03 u/sec=23.05	ppl=6.577669 
[2023-12-17:12:36:33:INFO:sockeye.training:__call__] E=26 B=104250	s/sec=2033.60 tok/sec=102017.93 u/sec=22.94	ppl=6.574667 
[2023-12-17:12:36:35:INFO:sockeye.training:__call__] E=26 B=104300	s/sec=2151.67 tok/sec=97301.92 u/sec=23.47	ppl=6.606225 
[2023-12-17:12:36:37:INFO:sockeye.training:__call__] E=26 B=104350	s/sec=2012.40 tok/sec=99975.00 u/sec=23.16	ppl=6.620446 
[2023-12-17:12:36:39:INFO:sockeye.training:__call__] E=26 B=104400	s/sec=2087.68 tok/sec=101405.82 u/sec=22.81	ppl=6.630073 
[2023-12-17:12:36:41:INFO:sockeye.training:__call__] E=26 B=104450	s/sec=2137.62 tok/sec=100467.97 u/sec=23.19	ppl=6.639066 
[2023-12-17:12:36:43:INFO:sockeye.training:__call__] E=26 B=104500	s/sec=2254.81 tok/sec=95385.40 u/sec=23.72	ppl=6.665162 
[2023-12-17:12:36:46:INFO:sockeye.training:__call__] E=26 B=104550	s/sec=1908.44 tok/sec=102105.36 u/sec=22.85	ppl=6.673099 
[2023-12-17:12:36:48:INFO:sockeye.training:__call__] E=26 B=104600	s/sec=2045.95 tok/sec=98921.28 u/sec=23.29	ppl=6.676880 
[2023-12-17:12:36:50:INFO:sockeye.training:__call__] E=26 B=104650	s/sec=2038.35 tok/sec=96804.29 u/sec=23.81	ppl=6.695172 
[2023-12-17:12:36:52:INFO:sockeye.training:__call__] E=26 B=104700	s/sec=2026.31 tok/sec=99974.80 u/sec=23.28	ppl=6.707066 
[2023-12-17:12:36:54:INFO:sockeye.training:__call__] E=26 B=104750	s/sec=1918.26 tok/sec=101591.52 u/sec=22.66	ppl=6.713886 
[2023-12-17:12:36:56:INFO:sockeye.training:__call__] E=26 B=104800	s/sec=2131.00 tok/sec=99481.90 u/sec=23.57	ppl=6.728059 
[2023-12-17:12:36:58:INFO:sockeye.training:__call__] E=26 B=104850	s/sec=2021.15 tok/sec=99919.83 u/sec=23.39	ppl=6.736957 
[2023-12-17:12:37:00:INFO:sockeye.training:__call__] E=26 B=104900	s/sec=2407.58 tok/sec=93160.49 u/sec=24.31	ppl=6.759378 
[2023-12-17:12:37:03:INFO:sockeye.training:__call__] E=26 B=104950	s/sec=1922.58 tok/sec=98940.56 u/sec=23.75	ppl=6.764101 
[2023-12-17:12:37:05:INFO:sockeye.training:__call__] E=26 B=105000	s/sec=2208.45 tok/sec=97050.45 u/sec=23.72	ppl=6.780808 
[2023-12-17:12:37:07:INFO:sockeye.training:__call__] E=26 B=105050	s/sec=2288.68 tok/sec=97931.23 u/sec=23.53	ppl=6.790344 
[2023-12-17:12:37:09:INFO:sockeye.training:__call__] E=26 B=105100	s/sec=2065.74 tok/sec=96976.25 u/sec=23.65	ppl=6.801885 
[2023-12-17:12:37:11:INFO:sockeye.training:__call__] E=26 B=105150	s/sec=2060.78 tok/sec=98499.11 u/sec=23.33	ppl=6.810818 
[2023-12-17:12:37:13:INFO:sockeye.training:__call__] E=26 B=105200	s/sec=2101.22 tok/sec=98571.70 u/sec=23.20	ppl=6.821416 
[2023-12-17:12:37:15:INFO:sockeye.training:__call__] E=26 B=105250	s/sec=2159.59 tok/sec=97240.86 u/sec=23.11	ppl=6.834068 
[2023-12-17:12:37:17:INFO:sockeye.training:__call__] E=26 B=105300	s/sec=1977.41 tok/sec=99981.58 u/sec=23.14	ppl=6.838683 
[2023-12-17:12:37:20:INFO:sockeye.training:__call__] E=26 B=105350	s/sec=1914.81 tok/sec=101084.36 u/sec=22.97	ppl=6.845588 
[2023-12-17:12:37:22:INFO:sockeye.training:__call__] E=26 B=105400	s/sec=2114.55 tok/sec=100376.62 u/sec=23.06	ppl=6.854170 
[2023-12-17:12:37:24:INFO:sockeye.training:__call__] E=26 B=105450	s/sec=2250.42 tok/sec=99447.27 u/sec=23.10	ppl=6.862777 
[2023-12-17:12:37:26:INFO:sockeye.training:__call__] E=26 B=105500	s/sec=1965.07 tok/sec=100735.69 u/sec=23.09	ppl=6.869077 
[2023-12-17:12:37:28:INFO:sockeye.training:__call__] E=26 B=105550	s/sec=1921.09 tok/sec=101172.37 u/sec=23.18	ppl=6.875622 
[2023-12-17:12:37:30:INFO:sockeye.training:__call__] E=26 B=105600	s/sec=2039.91 tok/sec=99616.42 u/sec=23.31	ppl=6.882252 
[2023-12-17:12:37:33:INFO:sockeye.training:__call__] E=26 B=105650	s/sec=1982.80 tok/sec=103237.18 u/sec=22.82	ppl=6.886545 
[2023-12-17:12:37:35:INFO:sockeye.training:__call__] E=26 B=105700	s/sec=2058.42 tok/sec=99691.00 u/sec=23.10	ppl=6.895727 
[2023-12-17:12:37:37:INFO:sockeye.training:__call__] E=26 B=105750	s/sec=1965.95 tok/sec=101197.13 u/sec=22.71	ppl=6.902095 
[2023-12-17:12:37:39:INFO:sockeye.training:__call__] E=26 B=105800	s/sec=1865.60 tok/sec=103108.97 u/sec=22.68	ppl=6.905478 
[2023-12-17:12:37:41:INFO:sockeye.training:__call__] E=26 B=105850	s/sec=2256.60 tok/sec=95854.03 u/sec=23.86	ppl=6.913436 
[2023-12-17:12:37:43:INFO:sockeye.training:__call__] E=26 B=105900	s/sec=2167.58 tok/sec=97884.85 u/sec=23.36	ppl=6.919695 
[2023-12-17:12:37:46:INFO:sockeye.training:__call__] E=26 B=105950	s/sec=1958.92 tok/sec=101878.52 u/sec=23.23	ppl=6.924835 
[2023-12-17:12:37:48:INFO:sockeye.training:__call__] E=26 B=106000	s/sec=2098.31 tok/sec=93726.90 u/sec=22.57	ppl=6.936285 
[2023-12-17:12:37:50:INFO:sockeye.training:__call__] E=26 B=106050	s/sec=2285.62 tok/sec=94614.19 u/sec=23.73	ppl=6.946259 
[2023-12-17:12:37:52:INFO:sockeye.training:__call__] E=26 B=106100	s/sec=2060.25 tok/sec=99701.12 u/sec=23.41	ppl=6.953595 
[2023-12-17:12:37:54:INFO:sockeye.training:__call__] E=26 B=106150	s/sec=1999.51 tok/sec=100893.88 u/sec=23.14	ppl=6.960082 
[2023-12-17:12:37:56:INFO:sockeye.training:__call__] E=26 B=106200	s/sec=2191.46 tok/sec=97193.91 u/sec=23.57	ppl=6.968863 
[2023-12-17:12:37:59:INFO:sockeye.training:__call__] E=26 B=106250	s/sec=2030.25 tok/sec=99034.43 u/sec=23.33	ppl=6.973890 
[2023-12-17:12:38:01:INFO:sockeye.training:__call__] E=26 B=106300	s/sec=2222.51 tok/sec=97036.12 u/sec=23.11	ppl=6.985160 
[2023-12-17:12:38:03:INFO:sockeye.training:__call__] E=26 B=106350	s/sec=2191.88 tok/sec=95530.38 u/sec=24.16	ppl=6.992612 
[2023-12-17:12:38:05:INFO:sockeye.training:__call__] E=26 B=106400	s/sec=2008.46 tok/sec=100045.87 u/sec=23.12	ppl=6.998074 
[2023-12-17:12:38:07:INFO:sockeye.training:__call__] E=26 B=106450	s/sec=2029.39 tok/sec=105825.86 u/sec=21.14	ppl=7.006570 
[2023-12-17:12:38:09:INFO:sockeye.training:__call__] E=26 B=106500	s/sec=2133.10 tok/sec=98802.54 u/sec=23.35	ppl=7.014931 
[2023-12-17:12:38:12:INFO:sockeye.training:__call__] E=26 B=106550	s/sec=2037.15 tok/sec=98784.77 u/sec=23.71	ppl=7.021293 
[2023-12-17:12:38:14:INFO:sockeye.training:__call__] E=26 B=106600	s/sec=2292.34 tok/sec=98057.89 u/sec=23.56	ppl=7.030231 
[2023-12-17:12:38:16:INFO:sockeye.training:__call__] E=26 B=106650	s/sec=2078.66 tok/sec=97663.21 u/sec=23.58	ppl=7.038260 
[2023-12-17:12:38:18:INFO:sockeye.training:__call__] E=26 B=106700	s/sec=2143.07 tok/sec=98596.24 u/sec=23.33	ppl=7.044373 
[2023-12-17:12:38:20:INFO:sockeye.training:__call__] E=26 B=106750	s/sec=2027.72 tok/sec=101060.21 u/sec=23.13	ppl=7.049528 
[2023-12-17:12:38:22:INFO:sockeye.training:__call__] E=26 B=106800	s/sec=2052.87 tok/sec=98143.78 u/sec=23.67	ppl=7.055532 
[2023-12-17:12:38:24:INFO:sockeye.training:__call__] E=26 B=106850	s/sec=2103.16 tok/sec=96580.72 u/sec=23.39	ppl=7.063876 
[2023-12-17:12:38:26:INFO:sockeye.training:__call__] E=26 B=106900	s/sec=2080.91 tok/sec=99404.86 u/sec=23.39	ppl=7.071274 
[2023-12-17:12:38:29:INFO:sockeye.training:__call__] E=26 B=106950	s/sec=2073.56 tok/sec=100468.81 u/sec=22.90	ppl=7.078738 
[2023-12-17:12:38:31:INFO:sockeye.training:__call__] E=26 B=107000	s/sec=2024.11 tok/sec=96923.89 u/sec=22.79	ppl=7.084885 
[2023-12-17:12:38:33:INFO:sockeye.training:__call__] E=26 B=107050	s/sec=2047.57 tok/sec=97550.64 u/sec=22.89	ppl=7.090173 
[2023-12-17:12:38:35:INFO:sockeye.training:__call__] E=26 B=107100	s/sec=2058.29 tok/sec=99241.52 u/sec=23.10	ppl=7.096807 
[2023-12-17:12:38:37:INFO:sockeye.training:__call__] E=26 B=107150	s/sec=2251.85 tok/sec=95059.62 u/sec=23.89	ppl=7.104901 
[2023-12-17:12:38:39:INFO:sockeye.training:__call__] E=26 B=107200	s/sec=1774.61 tok/sec=102934.87 u/sec=22.77	ppl=7.107197 
[2023-12-17:12:38:42:INFO:sockeye.training:__call__] E=26 B=107250	s/sec=2054.45 tok/sec=100382.98 u/sec=23.43	ppl=7.113836 
[2023-12-17:12:38:44:INFO:sockeye.training:__call__] E=26 B=107300	s/sec=2335.81 tok/sec=94448.10 u/sec=24.05	ppl=7.121506 
[2023-12-17:12:38:46:INFO:sockeye.training:__call__] E=26 B=107350	s/sec=1746.89 tok/sec=105848.58 u/sec=22.47	ppl=7.127065 
[2023-12-17:12:38:48:INFO:sockeye.training:__call__] E=26 B=107400	s/sec=2139.57 tok/sec=99156.04 u/sec=23.46	ppl=7.135621 
[2023-12-17:12:38:50:INFO:sockeye.training:__call__] E=26 B=107450	s/sec=2186.93 tok/sec=96632.73 u/sec=23.61	ppl=7.143041 
[2023-12-17:12:38:52:INFO:sockeye.training:__call__] E=26 B=107500	s/sec=1617.85 tok/sec=103855.23 u/sec=22.22	ppl=7.145230 
[2023-12-17:12:38:55:INFO:sockeye.training:__call__] E=26 B=107550	s/sec=2248.35 tok/sec=98480.73 u/sec=23.27	ppl=7.152001 
[2023-12-17:12:38:57:INFO:sockeye.training:__call__] E=26 B=107600	s/sec=2085.55 tok/sec=99189.21 u/sec=23.11	ppl=7.157626 
[2023-12-17:12:38:59:INFO:sockeye.training:__call__] E=26 B=107650	s/sec=1917.70 tok/sec=100937.70 u/sec=22.92	ppl=7.161759 
[2023-12-17:12:39:01:INFO:sockeye.training:__call__] E=26 B=107700	s/sec=2233.16 tok/sec=100021.48 u/sec=23.34	ppl=7.166726 
[2023-12-17:12:39:03:INFO:sockeye.training:__call__] E=26 B=107750	s/sec=2081.95 tok/sec=97558.55 u/sec=23.19	ppl=7.171582 
[2023-12-17:12:39:05:INFO:sockeye.training:__call__] E=26 B=107800	s/sec=1854.66 tok/sec=104672.51 u/sec=22.64	ppl=7.175567 
[2023-12-17:12:39:08:INFO:sockeye.training:__call__] E=26 B=107850	s/sec=1998.32 tok/sec=96003.75 u/sec=21.95	ppl=7.181196 
[2023-12-17:12:39:10:INFO:sockeye.training:__call__] E=27 B=107900	s/sec=1998.63 tok/sec=92764.63 u/sec=22.11	ppl=7.175098 
[2023-12-17:12:39:12:INFO:sockeye.training:__call__] E=27 B=107950	s/sec=1912.55 tok/sec=101427.48 u/sec=22.86	ppl=7.163458 
[2023-12-17:12:39:14:INFO:sockeye.training:__call__] E=27 B=108000	s/sec=1954.15 tok/sec=97217.88 u/sec=22.83	ppl=7.153931 
[2023-12-17:12:39:14:INFO:sockeye.training:_create_checkpoint] Checkpoint [27]	Updates=108000 Epoch=27 Samples=9624648 Time-cost=172.676 Updates/sec=23.165
[2023-12-17:12:39:14:INFO:sockeye.training:_create_checkpoint] Checkpoint [27]	Train-ppl=7.153931
[2023-12-17:12:39:17:INFO:sockeye.training:_evaluate] Checkpoint [27]	Validation-ppl=30.596437	Validation-bleu=0.165977	Validation-chrf=31.435920	Validation-rouge1=0.169933	Validation-rouge2=0.082795	Validation-rougel=0.163457	Validation-length-ratio-mse=1.255662	Validation-ter=93.809327	Validation-avg-sec-per-sent=0.003425	Validation-decode-walltime=1.712455
[2023-12-17:12:39:17:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 3 checkpoints, best so far: 32.780032
[2023-12-17:12:39:17:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:39:17:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [27]	Learning-rate=0.000162
[2023-12-17:12:39:18:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:39:18:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:39:18:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00027"
[2023-12-17:12:39:20:INFO:sockeye.training:__call__] E=27 B=108050	s/sec=816.12 tok/sec=36088.84 u/sec=8.76	ppl=6.420340 
[2023-12-17:12:39:22:INFO:sockeye.training:__call__] E=27 B=108100	s/sec=2018.84 tok/sec=99012.46 u/sec=23.19	ppl=6.417119 
[2023-12-17:12:39:24:INFO:sockeye.training:__call__] E=27 B=108150	s/sec=2333.31 tok/sec=92880.65 u/sec=23.52	ppl=6.472234 
[2023-12-17:12:39:27:INFO:sockeye.training:__call__] E=27 B=108200	s/sec=2151.54 tok/sec=96009.59 u/sec=22.79	ppl=6.476217 
[2023-12-17:12:39:29:INFO:sockeye.training:__call__] E=27 B=108250	s/sec=1690.06 tok/sec=105431.07 u/sec=22.47	ppl=6.445113 
[2023-12-17:12:39:31:INFO:sockeye.training:__call__] E=27 B=108300	s/sec=1992.02 tok/sec=99593.33 u/sec=23.18	ppl=6.448210 
[2023-12-17:12:39:33:INFO:sockeye.training:__call__] E=27 B=108350	s/sec=2257.92 tok/sec=94168.51 u/sec=23.33	ppl=6.476064 
[2023-12-17:12:39:35:INFO:sockeye.training:__call__] E=27 B=108400	s/sec=2057.13 tok/sec=101587.95 u/sec=22.92	ppl=6.467580 
[2023-12-17:12:39:37:INFO:sockeye.training:__call__] E=27 B=108450	s/sec=2041.83 tok/sec=97344.16 u/sec=22.55	ppl=6.478141 
[2023-12-17:12:39:40:INFO:sockeye.training:__call__] E=27 B=108500	s/sec=1785.19 tok/sec=97834.22 u/sec=22.86	ppl=6.476965 
[2023-12-17:12:39:42:INFO:sockeye.training:__call__] E=27 B=108550	s/sec=2148.47 tok/sec=93963.73 u/sec=23.39	ppl=6.491218 
[2023-12-17:12:39:44:INFO:sockeye.training:__call__] E=27 B=108600	s/sec=2004.76 tok/sec=98739.01 u/sec=23.08	ppl=6.502368 
[2023-12-17:12:39:46:INFO:sockeye.training:__call__] E=27 B=108650	s/sec=1887.63 tok/sec=102785.32 u/sec=22.60	ppl=6.505378 
[2023-12-17:12:39:48:INFO:sockeye.training:__call__] E=27 B=108700	s/sec=2053.16 tok/sec=99244.89 u/sec=23.55	ppl=6.513610 
[2023-12-17:12:39:50:INFO:sockeye.training:__call__] E=27 B=108750	s/sec=2115.33 tok/sec=99499.30 u/sec=23.40	ppl=6.522736 
[2023-12-17:12:39:53:INFO:sockeye.training:__call__] E=27 B=108800	s/sec=2016.43 tok/sec=98524.11 u/sec=22.23	ppl=6.531868 
[2023-12-17:12:39:55:INFO:sockeye.training:__call__] E=27 B=108850	s/sec=2041.29 tok/sec=98731.08 u/sec=23.41	ppl=6.541381 
[2023-12-17:12:39:57:INFO:sockeye.training:__call__] E=27 B=108900	s/sec=2024.48 tok/sec=100521.01 u/sec=23.13	ppl=6.550411 
[2023-12-17:12:39:59:INFO:sockeye.training:__call__] E=27 B=108950	s/sec=2261.42 tok/sec=94664.67 u/sec=24.00	ppl=6.566480 
[2023-12-17:12:40:01:INFO:sockeye.training:__call__] E=27 B=109000	s/sec=1955.61 tok/sec=99631.65 u/sec=23.33	ppl=6.574619 
[2023-12-17:12:40:03:INFO:sockeye.training:__call__] E=27 B=109050	s/sec=2172.78 tok/sec=94305.52 u/sec=23.29	ppl=6.587373 
[2023-12-17:12:40:06:INFO:sockeye.training:__call__] E=27 B=109100	s/sec=1983.52 tok/sec=101064.23 u/sec=22.87	ppl=6.597465 
[2023-12-17:12:40:08:INFO:sockeye.training:__call__] E=27 B=109150	s/sec=1980.19 tok/sec=100647.24 u/sec=22.54	ppl=6.605166 
[2023-12-17:12:40:10:INFO:sockeye.training:__call__] E=27 B=109200	s/sec=2018.38 tok/sec=98975.40 u/sec=23.36	ppl=6.617458 
[2023-12-17:12:40:12:INFO:sockeye.training:__call__] E=27 B=109250	s/sec=2032.27 tok/sec=99523.32 u/sec=22.68	ppl=6.624014 
[2023-12-17:12:40:14:INFO:sockeye.training:__call__] E=27 B=109300	s/sec=2255.01 tok/sec=96500.70 u/sec=24.01	ppl=6.637204 
[2023-12-17:12:40:16:INFO:sockeye.training:__call__] E=27 B=109350	s/sec=2086.20 tok/sec=102559.24 u/sec=22.79	ppl=6.642878 
[2023-12-17:12:40:19:INFO:sockeye.training:__call__] E=27 B=109400	s/sec=1895.62 tok/sec=102591.88 u/sec=23.01	ppl=6.652341 
[2023-12-17:12:40:21:INFO:sockeye.training:__call__] E=27 B=109450	s/sec=2024.19 tok/sec=101363.74 u/sec=22.96	ppl=6.662435 
[2023-12-17:12:40:23:INFO:sockeye.training:__call__] E=27 B=109500	s/sec=2298.73 tok/sec=95235.15 u/sec=23.55	ppl=6.674574 
[2023-12-17:12:40:25:INFO:sockeye.training:__call__] E=27 B=109550	s/sec=2114.16 tok/sec=99985.60 u/sec=23.35	ppl=6.684108 
[2023-12-17:12:40:27:INFO:sockeye.training:__call__] E=27 B=109600	s/sec=1999.85 tok/sec=99593.21 u/sec=22.68	ppl=6.694938 
[2023-12-17:12:40:29:INFO:sockeye.training:__call__] E=27 B=109650	s/sec=1779.33 tok/sec=102697.05 u/sec=23.17	ppl=6.697865 
[2023-12-17:12:40:31:INFO:sockeye.training:__call__] E=27 B=109700	s/sec=2187.03 tok/sec=96918.37 u/sec=23.41	ppl=6.709882 
[2023-12-17:12:40:34:INFO:sockeye.training:__call__] E=27 B=109750	s/sec=1846.24 tok/sec=102271.54 u/sec=22.99	ppl=6.715477 
[2023-12-17:12:40:36:INFO:sockeye.training:__call__] E=27 B=109800	s/sec=2246.85 tok/sec=97723.10 u/sec=23.40	ppl=6.723545 
[2023-12-17:12:40:38:INFO:sockeye.training:__call__] E=27 B=109850	s/sec=1777.46 tok/sec=104411.41 u/sec=22.40	ppl=6.728690 
[2023-12-17:12:40:40:INFO:sockeye.training:__call__] E=27 B=109900	s/sec=2307.67 tok/sec=98520.27 u/sec=23.34	ppl=6.740078 
[2023-12-17:12:40:42:INFO:sockeye.training:__call__] E=27 B=109950	s/sec=2111.35 tok/sec=100279.98 u/sec=23.11	ppl=6.749360 
[2023-12-17:12:40:44:INFO:sockeye.training:__call__] E=27 B=110000	s/sec=2075.30 tok/sec=97671.90 u/sec=23.71	ppl=6.756586 
[2023-12-17:12:40:47:INFO:sockeye.training:__call__] E=27 B=110050	s/sec=1993.38 tok/sec=103082.10 u/sec=21.33	ppl=6.762470 
[2023-12-17:12:40:49:INFO:sockeye.training:__call__] E=27 B=110100	s/sec=2227.36 tok/sec=98394.95 u/sec=23.52	ppl=6.771684 
[2023-12-17:12:40:51:INFO:sockeye.training:__call__] E=27 B=110150	s/sec=1886.82 tok/sec=100035.01 u/sec=23.26	ppl=6.778199 
[2023-12-17:12:40:53:INFO:sockeye.training:__call__] E=27 B=110200	s/sec=2044.01 tok/sec=99663.17 u/sec=22.03	ppl=6.787298 
[2023-12-17:12:40:55:INFO:sockeye.training:__call__] E=27 B=110250	s/sec=2170.32 tok/sec=98835.40 u/sec=23.31	ppl=6.797847 
[2023-12-17:12:40:58:INFO:sockeye.training:__call__] E=27 B=110300	s/sec=2155.65 tok/sec=99094.01 u/sec=22.95	ppl=6.804633 
[2023-12-17:12:41:00:INFO:sockeye.training:__call__] E=27 B=110350	s/sec=2131.81 tok/sec=100545.49 u/sec=22.81	ppl=6.815006 
[2023-12-17:12:41:02:INFO:sockeye.training:__call__] E=27 B=110400	s/sec=2259.36 tok/sec=95915.00 u/sec=23.65	ppl=6.824637 
[2023-12-17:12:41:04:INFO:sockeye.training:__call__] E=27 B=110450	s/sec=2084.30 tok/sec=97077.44 u/sec=23.43	ppl=6.832094 
[2023-12-17:12:41:06:INFO:sockeye.training:__call__] E=27 B=110500	s/sec=2168.19 tok/sec=98281.20 u/sec=23.32	ppl=6.839133 
[2023-12-17:12:41:08:INFO:sockeye.training:__call__] E=27 B=110550	s/sec=2070.61 tok/sec=102328.30 u/sec=23.19	ppl=6.842747 
[2023-12-17:12:41:11:INFO:sockeye.training:__call__] E=27 B=110600	s/sec=1827.65 tok/sec=103029.19 u/sec=23.12	ppl=6.846641 
[2023-12-17:12:41:13:INFO:sockeye.training:__call__] E=27 B=110650	s/sec=2056.77 tok/sec=99327.83 u/sec=23.54	ppl=6.851993 
[2023-12-17:12:41:15:INFO:sockeye.training:__call__] E=27 B=110700	s/sec=2126.88 tok/sec=98121.87 u/sec=23.49	ppl=6.858086 
[2023-12-17:12:41:17:INFO:sockeye.training:__call__] E=27 B=110750	s/sec=2073.23 tok/sec=99247.33 u/sec=23.22	ppl=6.865256 
[2023-12-17:12:41:19:INFO:sockeye.training:__call__] E=27 B=110800	s/sec=1825.89 tok/sec=102219.95 u/sec=23.24	ppl=6.869046 
[2023-12-17:12:41:21:INFO:sockeye.training:__call__] E=27 B=110850	s/sec=1874.89 tok/sec=103403.15 u/sec=22.19	ppl=6.871588 
[2023-12-17:12:41:24:INFO:sockeye.training:__call__] E=27 B=110900	s/sec=2174.96 tok/sec=99722.50 u/sec=23.12	ppl=6.879751 
[2023-12-17:12:41:26:INFO:sockeye.training:__call__] E=27 B=110950	s/sec=1920.45 tok/sec=102716.52 u/sec=22.86	ppl=6.884210 
[2023-12-17:12:41:28:INFO:sockeye.training:__call__] E=27 B=111000	s/sec=2193.62 tok/sec=90649.92 u/sec=23.16	ppl=6.891934 
[2023-12-17:12:41:30:INFO:sockeye.training:__call__] E=27 B=111050	s/sec=1905.64 tok/sec=98735.77 u/sec=23.26	ppl=6.898137 
[2023-12-17:12:41:32:INFO:sockeye.training:__call__] E=27 B=111100	s/sec=2304.97 tok/sec=97375.49 u/sec=23.93	ppl=6.905961 
[2023-12-17:12:41:34:INFO:sockeye.training:__call__] E=27 B=111150	s/sec=2058.56 tok/sec=98990.69 u/sec=23.44	ppl=6.911199 
[2023-12-17:12:41:36:INFO:sockeye.training:__call__] E=27 B=111200	s/sec=2318.28 tok/sec=97102.00 u/sec=23.71	ppl=6.918834 
[2023-12-17:12:41:39:INFO:sockeye.training:__call__] E=27 B=111250	s/sec=1902.16 tok/sec=97903.48 u/sec=22.52	ppl=6.925702 
[2023-12-17:12:41:41:INFO:sockeye.training:__call__] E=27 B=111300	s/sec=2059.10 tok/sec=100318.40 u/sec=23.15	ppl=6.931376 
[2023-12-17:12:41:43:INFO:sockeye.training:__call__] E=27 B=111350	s/sec=2179.83 tok/sec=97922.48 u/sec=23.61	ppl=6.936628 
[2023-12-17:12:41:45:INFO:sockeye.training:__call__] E=27 B=111400	s/sec=2118.56 tok/sec=96525.23 u/sec=22.67	ppl=6.943346 
[2023-12-17:12:41:47:INFO:sockeye.training:__call__] E=27 B=111450	s/sec=2322.43 tok/sec=95734.46 u/sec=23.83	ppl=6.952102 
[2023-12-17:12:41:49:INFO:sockeye.training:__call__] E=27 B=111500	s/sec=2130.89 tok/sec=98571.60 u/sec=23.57	ppl=6.958347 
[2023-12-17:12:41:51:INFO:sockeye.training:__call__] E=27 B=111550	s/sec=1828.40 tok/sec=102593.70 u/sec=22.72	ppl=6.960448 
[2023-12-17:12:41:54:INFO:sockeye.training:__call__] E=27 B=111600	s/sec=1948.87 tok/sec=99466.83 u/sec=23.25	ppl=6.964740 
[2023-12-17:12:41:56:INFO:sockeye.training:__call__] E=27 B=111650	s/sec=2014.62 tok/sec=97902.05 u/sec=23.45	ppl=6.970451 
[2023-12-17:12:41:58:INFO:sockeye.training:__call__] E=27 B=111700	s/sec=2133.64 tok/sec=96989.06 u/sec=23.81	ppl=6.975444 
[2023-12-17:12:42:00:INFO:sockeye.training:__call__] E=27 B=111750	s/sec=2012.89 tok/sec=100016.77 u/sec=23.08	ppl=6.979198 
[2023-12-17:12:42:02:INFO:sockeye.training:__call__] E=27 B=111800	s/sec=2211.07 tok/sec=97754.20 u/sec=23.54	ppl=6.985347 
[2023-12-17:12:42:04:INFO:sockeye.training:__call__] E=27 B=111850	s/sec=2200.44 tok/sec=96954.71 u/sec=23.51	ppl=6.992944 
[2023-12-17:12:42:07:INFO:sockeye.training:__call__] E=28 B=111900	s/sec=2126.44 tok/sec=91469.47 u/sec=22.56	ppl=6.987252 
[2023-12-17:12:42:09:INFO:sockeye.training:__call__] E=28 B=111950	s/sec=2129.07 tok/sec=100676.53 u/sec=23.26	ppl=6.978002 
[2023-12-17:12:42:11:INFO:sockeye.training:__call__] E=28 B=112000	s/sec=2115.91 tok/sec=98316.56 u/sec=23.49	ppl=6.968380 
[2023-12-17:12:42:11:INFO:sockeye.training:_create_checkpoint] Checkpoint [28]	Updates=112000 Epoch=28 Samples=9981944 Time-cost=172.861 Updates/sec=23.140
[2023-12-17:12:42:11:INFO:sockeye.training:_create_checkpoint] Checkpoint [28]	Train-ppl=6.968380
[2023-12-17:12:42:13:INFO:sockeye.training:_evaluate] Checkpoint [28]	Validation-ppl=31.389028	Validation-bleu=0.169759	Validation-chrf=31.224072	Validation-rouge1=0.167421	Validation-rouge2=0.083228	Validation-rougel=0.159678	Validation-length-ratio-mse=1.249069	Validation-ter=93.891870	Validation-avg-sec-per-sent=0.003415	Validation-decode-walltime=1.707390
[2023-12-17:12:42:13:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 4 checkpoints, best so far: 32.780032
[2023-12-17:12:42:13:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:42:13:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [28]	Learning-rate=0.000162
[2023-12-17:12:42:14:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:42:14:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:42:14:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00028"
[2023-12-17:12:42:17:INFO:sockeye.training:__call__] E=28 B=112050	s/sec=844.63 tok/sec=35000.50 u/sec=8.71	ppl=6.390150 
[2023-12-17:12:42:19:INFO:sockeye.training:__call__] E=28 B=112100	s/sec=1801.69 tok/sec=102366.69 u/sec=21.70	ppl=6.284698 
[2023-12-17:12:42:21:INFO:sockeye.training:__call__] E=28 B=112150	s/sec=1890.16 tok/sec=102819.06 u/sec=22.76	ppl=6.258243 
[2023-12-17:12:42:23:INFO:sockeye.training:__call__] E=28 B=112200	s/sec=1931.45 tok/sec=100139.16 u/sec=23.13	ppl=6.252724 
[2023-12-17:12:42:25:INFO:sockeye.training:__call__] E=28 B=112250	s/sec=2096.19 tok/sec=95729.84 u/sec=23.86	ppl=6.266773 
[2023-12-17:12:42:27:INFO:sockeye.training:__call__] E=28 B=112300	s/sec=2027.68 tok/sec=99513.36 u/sec=23.38	ppl=6.284350 
[2023-12-17:12:42:30:INFO:sockeye.training:__call__] E=28 B=112350	s/sec=2157.57 tok/sec=98923.04 u/sec=23.09	ppl=6.306647 
[2023-12-17:12:42:32:INFO:sockeye.training:__call__] E=28 B=112400	s/sec=2118.26 tok/sec=96925.37 u/sec=23.64	ppl=6.329383 
[2023-12-17:12:42:34:INFO:sockeye.training:__call__] E=28 B=112450	s/sec=2259.90 tok/sec=95928.60 u/sec=23.62	ppl=6.347202 
[2023-12-17:12:42:36:INFO:sockeye.training:__call__] E=28 B=112500	s/sec=2045.34 tok/sec=100340.16 u/sec=23.16	ppl=6.354440 
[2023-12-17:12:42:38:INFO:sockeye.training:__call__] E=28 B=112550	s/sec=2037.12 tok/sec=101464.72 u/sec=22.65	ppl=6.359123 
[2023-12-17:12:42:40:INFO:sockeye.training:__call__] E=28 B=112600	s/sec=2002.57 tok/sec=101131.42 u/sec=22.88	ppl=6.363429 
[2023-12-17:12:42:43:INFO:sockeye.training:__call__] E=28 B=112650	s/sec=2048.14 tok/sec=101587.86 u/sec=23.27	ppl=6.368392 
[2023-12-17:12:42:45:INFO:sockeye.training:__call__] E=28 B=112700	s/sec=2168.93 tok/sec=98274.27 u/sec=23.62	ppl=6.379703 
[2023-12-17:12:42:47:INFO:sockeye.training:__call__] E=28 B=112750	s/sec=1923.10 tok/sec=99956.70 u/sec=23.25	ppl=6.387251 
[2023-12-17:12:42:49:INFO:sockeye.training:__call__] E=28 B=112800	s/sec=1914.28 tok/sec=100442.62 u/sec=23.05	ppl=6.397944 
[2023-12-17:12:42:51:INFO:sockeye.training:__call__] E=28 B=112850	s/sec=2091.84 tok/sec=103321.32 u/sec=22.31	ppl=6.409883 
[2023-12-17:12:42:53:INFO:sockeye.training:__call__] E=28 B=112900	s/sec=1979.45 tok/sec=102458.83 u/sec=23.08	ppl=6.419558 
[2023-12-17:12:42:56:INFO:sockeye.training:__call__] E=28 B=112950	s/sec=2105.02 tok/sec=103596.21 u/sec=22.68	ppl=6.425047 
[2023-12-17:12:42:58:INFO:sockeye.training:__call__] E=28 B=113000	s/sec=2365.92 tok/sec=93740.04 u/sec=24.16	ppl=6.437643 
[2023-12-17:12:43:00:INFO:sockeye.training:__call__] E=28 B=113050	s/sec=1866.56 tok/sec=100482.09 u/sec=23.24	ppl=6.446336 
[2023-12-17:12:43:02:INFO:sockeye.training:__call__] E=28 B=113100	s/sec=2030.03 tok/sec=99213.32 u/sec=23.07	ppl=6.456117 
[2023-12-17:12:43:04:INFO:sockeye.training:__call__] E=28 B=113150	s/sec=1851.61 tok/sec=101520.50 u/sec=22.60	ppl=6.459899 
[2023-12-17:12:43:06:INFO:sockeye.training:__call__] E=28 B=113200	s/sec=1735.59 tok/sec=93590.32 u/sec=21.65	ppl=6.464961 
[2023-12-17:12:43:09:INFO:sockeye.training:__call__] E=28 B=113250	s/sec=2266.82 tok/sec=85315.38 u/sec=22.38	ppl=6.482585 
[2023-12-17:12:43:11:INFO:sockeye.training:__call__] E=28 B=113300	s/sec=1858.69 tok/sec=98952.42 u/sec=21.92	ppl=6.490782 
[2023-12-17:12:43:13:INFO:sockeye.training:__call__] E=28 B=113350	s/sec=2349.26 tok/sec=94526.75 u/sec=24.15	ppl=6.501230 
[2023-12-17:12:43:15:INFO:sockeye.training:__call__] E=28 B=113400	s/sec=1715.40 tok/sec=102032.94 u/sec=22.11	ppl=6.506031 
[2023-12-17:12:43:18:INFO:sockeye.training:__call__] E=28 B=113450	s/sec=2053.25 tok/sec=88665.90 u/sec=22.40	ppl=6.516619 
[2023-12-17:12:43:20:INFO:sockeye.training:__call__] E=28 B=113500	s/sec=1948.44 tok/sec=92538.36 u/sec=22.02	ppl=6.525860 
[2023-12-17:12:43:22:INFO:sockeye.training:__call__] E=28 B=113550	s/sec=1606.59 tok/sec=104591.53 u/sec=22.67	ppl=6.527780 
[2023-12-17:12:43:24:INFO:sockeye.training:__call__] E=28 B=113600	s/sec=2177.98 tok/sec=96024.58 u/sec=23.27	ppl=6.538975 
[2023-12-17:12:43:26:INFO:sockeye.training:__call__] E=28 B=113650	s/sec=2156.73 tok/sec=96098.47 u/sec=23.48	ppl=6.548520 
[2023-12-17:12:43:28:INFO:sockeye.training:__call__] E=28 B=113700	s/sec=2087.48 tok/sec=100169.03 u/sec=23.42	ppl=6.554602 
[2023-12-17:12:43:31:INFO:sockeye.training:__call__] E=28 B=113750	s/sec=2087.62 tok/sec=101334.55 u/sec=22.61	ppl=6.563680 
[2023-12-17:12:43:33:INFO:sockeye.training:__call__] E=28 B=113800	s/sec=2179.58 tok/sec=98428.92 u/sec=23.37	ppl=6.574180 
[2023-12-17:12:43:35:INFO:sockeye.training:__call__] E=28 B=113850	s/sec=2165.46 tok/sec=97193.32 u/sec=23.54	ppl=6.583793 
[2023-12-17:12:43:37:INFO:sockeye.training:__call__] E=28 B=113900	s/sec=2236.86 tok/sec=96113.45 u/sec=23.58	ppl=6.591457 
[2023-12-17:12:43:39:INFO:sockeye.training:__call__] E=28 B=113950	s/sec=1872.99 tok/sec=105455.93 u/sec=22.21	ppl=6.595672 
[2023-12-17:12:43:42:INFO:sockeye.training:__call__] E=28 B=114000	s/sec=2065.23 tok/sec=98287.49 u/sec=22.72	ppl=6.602606 
[2023-12-17:12:43:44:INFO:sockeye.training:__call__] E=28 B=114050	s/sec=1925.00 tok/sec=100615.14 u/sec=23.00	ppl=6.607504 
[2023-12-17:12:43:46:INFO:sockeye.training:__call__] E=28 B=114100	s/sec=2164.52 tok/sec=95890.40 u/sec=24.24	ppl=6.612988 
[2023-12-17:12:43:48:INFO:sockeye.training:__call__] E=28 B=114150	s/sec=1814.63 tok/sec=100264.51 u/sec=23.00	ppl=6.615848 
[2023-12-17:12:43:50:INFO:sockeye.training:__call__] E=28 B=114200	s/sec=2022.42 tok/sec=101353.58 u/sec=22.73	ppl=6.622380 
[2023-12-17:12:43:52:INFO:sockeye.training:__call__] E=28 B=114250	s/sec=1688.40 tok/sec=104151.98 u/sec=22.94	ppl=6.626060 
[2023-12-17:12:43:54:INFO:sockeye.training:__call__] E=28 B=114300	s/sec=2207.46 tok/sec=99638.26 u/sec=23.07	ppl=6.634538 
[2023-12-17:12:43:57:INFO:sockeye.training:__call__] E=28 B=114350	s/sec=2131.05 tok/sec=98013.06 u/sec=23.53	ppl=6.642004 
[2023-12-17:12:43:59:INFO:sockeye.training:__call__] E=28 B=114400	s/sec=2245.23 tok/sec=98834.74 u/sec=23.35	ppl=6.651790 
[2023-12-17:12:44:01:INFO:sockeye.training:__call__] E=28 B=114450	s/sec=2207.88 tok/sec=98533.11 u/sec=23.55	ppl=6.658938 
[2023-12-17:12:44:03:INFO:sockeye.training:__call__] E=28 B=114500	s/sec=2042.32 tok/sec=99955.23 u/sec=23.12	ppl=6.665036 
[2023-12-17:12:44:05:INFO:sockeye.training:__call__] E=28 B=114550	s/sec=2218.65 tok/sec=95520.73 u/sec=23.95	ppl=6.671182 
[2023-12-17:12:44:07:INFO:sockeye.training:__call__] E=28 B=114600	s/sec=2071.19 tok/sec=99269.81 u/sec=22.99	ppl=6.677129 
[2023-12-17:12:44:10:INFO:sockeye.training:__call__] E=28 B=114650	s/sec=1773.13 tok/sec=105402.58 u/sec=21.99	ppl=6.682421 
[2023-12-17:12:44:12:INFO:sockeye.training:__call__] E=28 B=114700	s/sec=1995.69 tok/sec=97818.20 u/sec=22.93	ppl=6.689759 
[2023-12-17:12:44:14:INFO:sockeye.training:__call__] E=28 B=114750	s/sec=2150.80 tok/sec=96367.87 u/sec=23.75	ppl=6.696263 
[2023-12-17:12:44:16:INFO:sockeye.training:__call__] E=28 B=114800	s/sec=2097.16 tok/sec=99016.01 u/sec=23.41	ppl=6.701942 
[2023-12-17:12:44:18:INFO:sockeye.training:__call__] E=28 B=114850	s/sec=2141.80 tok/sec=99338.95 u/sec=22.57	ppl=6.708252 
[2023-12-17:12:44:20:INFO:sockeye.training:__call__] E=28 B=114900	s/sec=1921.55 tok/sec=100113.04 u/sec=23.14	ppl=6.713855 
[2023-12-17:12:44:23:INFO:sockeye.training:__call__] E=28 B=114950	s/sec=2019.44 tok/sec=99279.11 u/sec=23.20	ppl=6.720485 
[2023-12-17:12:44:25:INFO:sockeye.training:__call__] E=28 B=115000	s/sec=2204.58 tok/sec=97001.33 u/sec=24.00	ppl=6.727868 
[2023-12-17:12:44:27:INFO:sockeye.training:__call__] E=28 B=115050	s/sec=2131.54 tok/sec=97657.75 u/sec=23.62	ppl=6.735243 
[2023-12-17:12:44:29:INFO:sockeye.training:__call__] E=28 B=115100	s/sec=2099.42 tok/sec=100213.40 u/sec=22.98	ppl=6.742635 
[2023-12-17:12:44:31:INFO:sockeye.training:__call__] E=28 B=115150	s/sec=2034.27 tok/sec=100993.68 u/sec=22.95	ppl=6.749894 
[2023-12-17:12:44:33:INFO:sockeye.training:__call__] E=28 B=115200	s/sec=2012.92 tok/sec=99690.24 u/sec=23.52	ppl=6.754185 
[2023-12-17:12:44:35:INFO:sockeye.training:__call__] E=28 B=115250	s/sec=2324.31 tok/sec=96413.61 u/sec=23.58	ppl=6.761500 
[2023-12-17:12:44:38:INFO:sockeye.training:__call__] E=28 B=115300	s/sec=2096.07 tok/sec=99262.64 u/sec=22.90	ppl=6.766438 
[2023-12-17:12:44:40:INFO:sockeye.training:__call__] E=28 B=115350	s/sec=1942.54 tok/sec=102738.57 u/sec=22.48	ppl=6.771331 
[2023-12-17:12:44:42:INFO:sockeye.training:__call__] E=28 B=115400	s/sec=1965.22 tok/sec=100192.93 u/sec=23.13	ppl=6.775410 
[2023-12-17:12:44:44:INFO:sockeye.training:__call__] E=28 B=115450	s/sec=1976.51 tok/sec=99509.29 u/sec=23.22	ppl=6.779908 
[2023-12-17:12:44:46:INFO:sockeye.training:__call__] E=28 B=115500	s/sec=2188.35 tok/sec=98617.51 u/sec=23.34	ppl=6.784885 
[2023-12-17:12:44:48:INFO:sockeye.training:__call__] E=28 B=115550	s/sec=1993.07 tok/sec=99394.70 u/sec=23.11	ppl=6.789380 
[2023-12-17:12:44:50:INFO:sockeye.training:__call__] E=28 B=115600	s/sec=2113.27 tok/sec=98270.67 u/sec=23.34	ppl=6.795344 
[2023-12-17:12:44:53:INFO:sockeye.training:__call__] E=28 B=115650	s/sec=2284.26 tok/sec=97251.84 u/sec=23.44	ppl=6.804172 
[2023-12-17:12:44:55:INFO:sockeye.training:__call__] E=28 B=115700	s/sec=2223.50 tok/sec=96311.24 u/sec=23.80	ppl=6.809891 
[2023-12-17:12:44:57:INFO:sockeye.training:__call__] E=28 B=115750	s/sec=2214.29 tok/sec=97504.17 u/sec=23.58	ppl=6.816283 
[2023-12-17:12:44:59:INFO:sockeye.training:__call__] E=28 B=115800	s/sec=2028.09 tok/sec=101677.05 u/sec=23.00	ppl=6.821279 
[2023-12-17:12:45:01:INFO:sockeye.training:__call__] E=28 B=115850	s/sec=2264.88 tok/sec=99294.89 u/sec=23.44	ppl=6.827749 
[2023-12-17:12:45:03:INFO:sockeye.training:__call__] E=29 B=115900	s/sec=2005.45 tok/sec=91315.87 u/sec=23.00	ppl=6.820566 
[2023-12-17:12:45:05:INFO:sockeye.training:__call__] E=29 B=115950	s/sec=1844.63 tok/sec=101584.10 u/sec=23.10	ppl=6.809538 
[2023-12-17:12:45:08:INFO:sockeye.training:__call__] E=29 B=116000	s/sec=2129.01 tok/sec=99025.02 u/sec=23.34	ppl=6.800896 
[2023-12-17:12:45:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [29]	Updates=116000 Epoch=29 Samples=10337600 Time-cost=173.206 Updates/sec=23.094
[2023-12-17:12:45:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [29]	Train-ppl=6.800896
[2023-12-17:12:45:10:INFO:sockeye.training:_evaluate] Checkpoint [29]	Validation-ppl=32.048300	Validation-bleu=0.155628	Validation-chrf=30.123067	Validation-rouge1=0.158398	Validation-rouge2=0.075613	Validation-rougel=0.151485	Validation-length-ratio-mse=1.324581	Validation-ter=97.688816	Validation-avg-sec-per-sent=0.003493	Validation-decode-walltime=1.746609
[2023-12-17:12:45:10:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 5 checkpoints, best so far: 32.780032
[2023-12-17:12:45:10:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:45:10:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [29]	Learning-rate=0.000162
[2023-12-17:12:45:11:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:45:11:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=5, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:45:11:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00029"
[2023-12-17:12:45:13:INFO:sockeye.training:__call__] E=29 B=116050	s/sec=766.08 tok/sec=39830.68 u/sec=8.95	ppl=6.105870 
[2023-12-17:12:45:15:INFO:sockeye.training:__call__] E=29 B=116100	s/sec=2052.58 tok/sec=101172.44 u/sec=22.99	ppl=6.113376 
[2023-12-17:12:45:18:INFO:sockeye.training:__call__] E=29 B=116150	s/sec=2284.98 tok/sec=97613.59 u/sec=23.68	ppl=6.139344 
[2023-12-17:12:45:20:INFO:sockeye.training:__call__] E=29 B=116200	s/sec=2000.07 tok/sec=100728.10 u/sec=22.65	ppl=6.152866 
[2023-12-17:12:45:22:INFO:sockeye.training:__call__] E=29 B=116250	s/sec=2131.65 tok/sec=95314.52 u/sec=23.09	ppl=6.174623 
[2023-12-17:12:45:24:INFO:sockeye.training:__call__] E=29 B=116300	s/sec=1906.23 tok/sec=101895.25 u/sec=22.44	ppl=6.182109 
[2023-12-17:12:45:26:INFO:sockeye.training:__call__] E=29 B=116350	s/sec=2361.86 tok/sec=88985.61 u/sec=24.64	ppl=6.201161 
[2023-12-17:12:45:28:INFO:sockeye.training:__call__] E=29 B=116400	s/sec=2129.90 tok/sec=97757.38 u/sec=23.48	ppl=6.226717 
[2023-12-17:12:45:30:INFO:sockeye.training:__call__] E=29 B=116450	s/sec=1876.65 tok/sec=102489.09 u/sec=22.60	ppl=6.224577 
[2023-12-17:12:45:33:INFO:sockeye.training:__call__] E=29 B=116500	s/sec=2098.13 tok/sec=97922.68 u/sec=23.17	ppl=6.227758 
[2023-12-17:12:45:35:INFO:sockeye.training:__call__] E=29 B=116550	s/sec=2029.31 tok/sec=98983.69 u/sec=22.81	ppl=6.238108 
[2023-12-17:12:45:37:INFO:sockeye.training:__call__] E=29 B=116600	s/sec=2138.57 tok/sec=91357.13 u/sec=23.78	ppl=6.246993 
[2023-12-17:12:45:39:INFO:sockeye.training:__call__] E=29 B=116650	s/sec=2109.84 tok/sec=97872.85 u/sec=23.30	ppl=6.256195 
[2023-12-17:12:45:41:INFO:sockeye.training:__call__] E=29 B=116700	s/sec=1939.39 tok/sec=100298.27 u/sec=22.61	ppl=6.262035 
[2023-12-17:12:45:44:INFO:sockeye.training:__call__] E=29 B=116750	s/sec=1945.65 tok/sec=96830.79 u/sec=22.39	ppl=6.266781 
[2023-12-17:12:45:46:INFO:sockeye.training:__call__] E=29 B=116800	s/sec=2149.75 tok/sec=93123.16 u/sec=23.61	ppl=6.283012 
[2023-12-17:12:45:48:INFO:sockeye.training:__call__] E=29 B=116850	s/sec=2412.74 tok/sec=94732.50 u/sec=23.94	ppl=6.297564 
[2023-12-17:12:45:50:INFO:sockeye.training:__call__] E=29 B=116900	s/sec=1895.69 tok/sec=101783.50 u/sec=23.10	ppl=6.306373 
[2023-12-17:12:45:52:INFO:sockeye.training:__call__] E=29 B=116950	s/sec=2058.04 tok/sec=97937.08 u/sec=23.69	ppl=6.314245 
[2023-12-17:12:45:54:INFO:sockeye.training:__call__] E=29 B=117000	s/sec=2413.69 tok/sec=95254.05 u/sec=24.06	ppl=6.329865 
[2023-12-17:12:45:56:INFO:sockeye.training:__call__] E=29 B=117050	s/sec=2309.59 tok/sec=96610.21 u/sec=23.59	ppl=6.342876 
[2023-12-17:12:45:58:INFO:sockeye.training:__call__] E=29 B=117100	s/sec=2338.93 tok/sec=94841.22 u/sec=23.89	ppl=6.353458 
[2023-12-17:12:46:00:INFO:sockeye.training:__call__] E=29 B=117150	s/sec=1960.43 tok/sec=99001.73 u/sec=23.21	ppl=6.362545 
[2023-12-17:12:46:03:INFO:sockeye.training:__call__] E=29 B=117200	s/sec=1909.58 tok/sec=101981.89 u/sec=23.04	ppl=6.367592 
[2023-12-17:12:46:05:INFO:sockeye.training:__call__] E=29 B=117250	s/sec=2369.40 tok/sec=93850.02 u/sec=24.12	ppl=6.378014 
[2023-12-17:12:46:07:INFO:sockeye.training:__call__] E=29 B=117300	s/sec=2045.14 tok/sec=96419.22 u/sec=23.54	ppl=6.388227 
[2023-12-17:12:46:09:INFO:sockeye.training:__call__] E=29 B=117350	s/sec=1992.62 tok/sec=101066.16 u/sec=22.77	ppl=6.396400 
[2023-12-17:12:46:11:INFO:sockeye.training:__call__] E=29 B=117400	s/sec=2158.50 tok/sec=94517.96 u/sec=23.75	ppl=6.404399 
[2023-12-17:12:46:13:INFO:sockeye.training:__call__] E=29 B=117450	s/sec=2077.80 tok/sec=98450.94 u/sec=23.31	ppl=6.409936 
[2023-12-17:12:46:15:INFO:sockeye.training:__call__] E=29 B=117500	s/sec=1787.22 tok/sec=102973.33 u/sec=22.80	ppl=6.411826 
[2023-12-17:12:46:18:INFO:sockeye.training:__call__] E=29 B=117550	s/sec=1910.75 tok/sec=99373.66 u/sec=22.66	ppl=6.416108 
[2023-12-17:12:46:20:INFO:sockeye.training:__call__] E=29 B=117600	s/sec=2149.36 tok/sec=96758.43 u/sec=23.40	ppl=6.425193 
[2023-12-17:12:46:22:INFO:sockeye.training:__call__] E=29 B=117650	s/sec=1864.58 tok/sec=103076.57 u/sec=22.76	ppl=6.428604 
[2023-12-17:12:46:24:INFO:sockeye.training:__call__] E=29 B=117700	s/sec=1994.46 tok/sec=101654.34 u/sec=23.13	ppl=6.434470 
[2023-12-17:12:46:26:INFO:sockeye.training:__call__] E=29 B=117750	s/sec=2062.60 tok/sec=97980.85 u/sec=23.52	ppl=6.443377 
[2023-12-17:12:46:28:INFO:sockeye.training:__call__] E=29 B=117800	s/sec=2028.55 tok/sec=99014.46 u/sec=23.79	ppl=6.450812 
[2023-12-17:12:46:31:INFO:sockeye.training:__call__] E=29 B=117850	s/sec=1869.82 tok/sec=101903.55 u/sec=23.33	ppl=6.456471 
[2023-12-17:12:46:33:INFO:sockeye.training:__call__] E=29 B=117900	s/sec=1857.92 tok/sec=104160.04 u/sec=22.77	ppl=6.461676 
[2023-12-17:12:46:35:INFO:sockeye.training:__call__] E=29 B=117950	s/sec=1974.02 tok/sec=104245.07 u/sec=22.07	ppl=6.468176 
[2023-12-17:12:46:37:INFO:sockeye.training:__call__] E=29 B=118000	s/sec=2128.87 tok/sec=99062.52 u/sec=23.02	ppl=6.474436 
[2023-12-17:12:46:39:INFO:sockeye.training:__call__] E=29 B=118050	s/sec=1971.45 tok/sec=97919.25 u/sec=22.95	ppl=6.479503 
[2023-12-17:12:46:42:INFO:sockeye.training:__call__] E=29 B=118100	s/sec=1934.18 tok/sec=101754.05 u/sec=22.98	ppl=6.486206 
[2023-12-17:12:46:44:INFO:sockeye.training:__call__] E=29 B=118150	s/sec=2144.18 tok/sec=97849.14 u/sec=23.31	ppl=6.492030 
[2023-12-17:12:46:46:INFO:sockeye.training:__call__] E=29 B=118200	s/sec=1809.76 tok/sec=103546.98 u/sec=22.62	ppl=6.495159 
[2023-12-17:12:46:48:INFO:sockeye.training:__call__] E=29 B=118250	s/sec=2175.35 tok/sec=95776.67 u/sec=23.94	ppl=6.500696 
[2023-12-17:12:46:50:INFO:sockeye.training:__call__] E=29 B=118300	s/sec=1859.53 tok/sec=101527.64 u/sec=22.88	ppl=6.504648 
[2023-12-17:12:46:52:INFO:sockeye.training:__call__] E=29 B=118350	s/sec=1959.89 tok/sec=100698.06 u/sec=22.90	ppl=6.510634 
[2023-12-17:12:46:55:INFO:sockeye.training:__call__] E=29 B=118400	s/sec=1990.97 tok/sec=103398.78 u/sec=22.83	ppl=6.516478 
[2023-12-17:12:46:57:INFO:sockeye.training:__call__] E=29 B=118450	s/sec=1966.57 tok/sec=101701.90 u/sec=23.02	ppl=6.521527 
[2023-12-17:12:46:59:INFO:sockeye.training:__call__] E=29 B=118500	s/sec=1970.19 tok/sec=101714.50 u/sec=22.97	ppl=6.526429 
[2023-12-17:12:47:01:INFO:sockeye.training:__call__] E=29 B=118550	s/sec=2100.77 tok/sec=98887.42 u/sec=23.08	ppl=6.532284 
[2023-12-17:12:47:03:INFO:sockeye.training:__call__] E=29 B=118600	s/sec=2231.35 tok/sec=96891.56 u/sec=23.40	ppl=6.537941 
[2023-12-17:12:47:05:INFO:sockeye.training:__call__] E=29 B=118650	s/sec=2089.12 tok/sec=99322.62 u/sec=23.32	ppl=6.545142 
[2023-12-17:12:47:08:INFO:sockeye.training:__call__] E=29 B=118700	s/sec=1882.24 tok/sec=104723.52 u/sec=22.20	ppl=6.549732 
[2023-12-17:12:47:10:INFO:sockeye.training:__call__] E=29 B=118750	s/sec=2090.19 tok/sec=98762.22 u/sec=23.54	ppl=6.556588 
[2023-12-17:12:47:12:INFO:sockeye.training:__call__] E=29 B=118800	s/sec=2286.60 tok/sec=99354.33 u/sec=23.09	ppl=6.564652 
[2023-12-17:12:47:14:INFO:sockeye.training:__call__] E=29 B=118850	s/sec=2141.35 tok/sec=101548.50 u/sec=23.00	ppl=6.570187 
[2023-12-17:12:47:16:INFO:sockeye.training:__call__] E=29 B=118900	s/sec=2061.37 tok/sec=99868.51 u/sec=23.26	ppl=6.575462 
[2023-12-17:12:47:18:INFO:sockeye.training:__call__] E=29 B=118950	s/sec=2236.91 tok/sec=95623.95 u/sec=23.78	ppl=6.582277 
[2023-12-17:12:47:20:INFO:sockeye.training:__call__] E=29 B=119000	s/sec=1968.39 tok/sec=98495.13 u/sec=23.57	ppl=6.586427 
[2023-12-17:12:47:23:INFO:sockeye.training:__call__] E=29 B=119050	s/sec=2193.92 tok/sec=96234.93 u/sec=23.24	ppl=6.592900 
[2023-12-17:12:47:25:INFO:sockeye.training:__call__] E=29 B=119100	s/sec=2099.46 tok/sec=96473.26 u/sec=22.70	ppl=6.599729 
[2023-12-17:12:47:27:INFO:sockeye.training:__call__] E=29 B=119150	s/sec=2213.25 tok/sec=97047.09 u/sec=23.85	ppl=6.607356 
[2023-12-17:12:47:29:INFO:sockeye.training:__call__] E=29 B=119200	s/sec=2196.48 tok/sec=99167.96 u/sec=23.19	ppl=6.613992 
[2023-12-17:12:47:31:INFO:sockeye.training:__call__] E=29 B=119250	s/sec=2183.07 tok/sec=98483.13 u/sec=23.57	ppl=6.620188 
[2023-12-17:12:47:33:INFO:sockeye.training:__call__] E=29 B=119300	s/sec=2081.13 tok/sec=98738.95 u/sec=23.14	ppl=6.625070 
[2023-12-17:12:47:35:INFO:sockeye.training:__call__] E=29 B=119350	s/sec=2053.06 tok/sec=99758.85 u/sec=23.67	ppl=6.630705 
[2023-12-17:12:47:38:INFO:sockeye.training:__call__] E=29 B=119400	s/sec=2029.77 tok/sec=100971.88 u/sec=23.07	ppl=6.635158 
[2023-12-17:12:47:40:INFO:sockeye.training:__call__] E=29 B=119450	s/sec=2415.68 tok/sec=96777.88 u/sec=23.52	ppl=6.642558 
[2023-12-17:12:47:42:INFO:sockeye.training:__call__] E=29 B=119500	s/sec=1948.18 tok/sec=101924.76 u/sec=22.51	ppl=6.645057 
[2023-12-17:12:47:44:INFO:sockeye.training:__call__] E=29 B=119550	s/sec=2224.10 tok/sec=98593.12 u/sec=23.36	ppl=6.651982 
[2023-12-17:12:47:46:INFO:sockeye.training:__call__] E=29 B=119600	s/sec=2173.54 tok/sec=95756.51 u/sec=23.58	ppl=6.658127 
[2023-12-17:12:47:48:INFO:sockeye.training:__call__] E=29 B=119650	s/sec=2069.03 tok/sec=103182.31 u/sec=22.73	ppl=6.663028 
[2023-12-17:12:47:51:INFO:sockeye.training:__call__] E=29 B=119700	s/sec=1922.52 tok/sec=101537.63 u/sec=22.46	ppl=6.667300 
[2023-12-17:12:47:53:INFO:sockeye.training:__call__] E=29 B=119750	s/sec=2094.47 tok/sec=98063.36 u/sec=23.54	ppl=6.671864 
[2023-12-17:12:47:55:INFO:sockeye.training:__call__] E=29 B=119800	s/sec=1988.37 tok/sec=101247.70 u/sec=23.14	ppl=6.675618 
[2023-12-17:12:47:57:INFO:sockeye.training:__call__] E=29 B=119850	s/sec=1962.41 tok/sec=108560.96 u/sec=21.71	ppl=6.679838 
[2023-12-17:12:48:00:INFO:sockeye.training:__call__] E=30 B=119900	s/sec=1871.32 tok/sec=98461.33 u/sec=21.70	ppl=6.669639 
[2023-12-17:12:48:02:INFO:sockeye.training:__call__] E=30 B=119950	s/sec=2121.03 tok/sec=94808.63 u/sec=23.18	ppl=6.661524 
[2023-12-17:12:48:04:INFO:sockeye.training:__call__] E=30 B=120000	s/sec=1862.97 tok/sec=103146.50 u/sec=22.48	ppl=6.652088 
[2023-12-17:12:48:04:INFO:sockeye.training:_create_checkpoint] Checkpoint [30]	Updates=120000 Epoch=30 Samples=10694232 Time-cost=172.880 Updates/sec=23.137
[2023-12-17:12:48:04:INFO:sockeye.training:_create_checkpoint] Checkpoint [30]	Train-ppl=6.652088
[2023-12-17:12:48:07:INFO:sockeye.training:_evaluate] Checkpoint [30]	Validation-ppl=33.073982	Validation-bleu=0.169554	Validation-chrf=31.937972	Validation-rouge1=0.167111	Validation-rouge2=0.081628	Validation-rougel=0.159600	Validation-length-ratio-mse=1.325885	Validation-ter=97.977714	Validation-avg-sec-per-sent=0.003596	Validation-decode-walltime=1.798089
[2023-12-17:12:48:07:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 6 checkpoints, best so far: 32.780032
[2023-12-17:12:48:07:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:48:07:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [30]	Learning-rate=0.000162
[2023-12-17:12:48:07:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:48:07:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=6, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:48:08:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00030"
[2023-12-17:12:48:10:INFO:sockeye.training:__call__] E=30 B=120050	s/sec=764.25 tok/sec=35218.20 u/sec=8.54	ppl=6.025048 
[2023-12-17:12:48:12:INFO:sockeye.training:__call__] E=30 B=120100	s/sec=1848.99 tok/sec=104541.08 u/sec=22.93	ppl=6.004863 
[2023-12-17:12:48:14:INFO:sockeye.training:__call__] E=30 B=120150	s/sec=1948.24 tok/sec=99164.45 u/sec=23.11	ppl=6.005783 
[2023-12-17:12:48:16:INFO:sockeye.training:__call__] E=30 B=120200	s/sec=2079.07 tok/sec=98632.89 u/sec=23.29	ppl=6.038519 
[2023-12-17:12:48:18:INFO:sockeye.training:__call__] E=30 B=120250	s/sec=2072.86 tok/sec=99076.12 u/sec=23.51	ppl=6.044085 
[2023-12-17:12:48:21:INFO:sockeye.training:__call__] E=30 B=120300	s/sec=2069.10 tok/sec=101315.24 u/sec=23.30	ppl=6.053422 
[2023-12-17:12:48:23:INFO:sockeye.training:__call__] E=30 B=120350	s/sec=2340.78 tok/sec=95698.38 u/sec=23.75	ppl=6.073085 
[2023-12-17:12:48:25:INFO:sockeye.training:__call__] E=30 B=120400	s/sec=2046.26 tok/sec=98220.31 u/sec=23.38	ppl=6.072530 
[2023-12-17:12:48:27:INFO:sockeye.training:__call__] E=30 B=120450	s/sec=2199.50 tok/sec=97431.30 u/sec=23.74	ppl=6.085159 
[2023-12-17:12:48:29:INFO:sockeye.training:__call__] E=30 B=120500	s/sec=2428.55 tok/sec=95159.57 u/sec=23.83	ppl=6.110693 
[2023-12-17:12:48:31:INFO:sockeye.training:__call__] E=30 B=120550	s/sec=1821.63 tok/sec=103767.10 u/sec=22.86	ppl=6.108706 
[2023-12-17:12:48:33:INFO:sockeye.training:__call__] E=30 B=120600	s/sec=2214.60 tok/sec=97322.45 u/sec=23.46	ppl=6.118779 
[2023-12-17:12:48:35:INFO:sockeye.training:__call__] E=30 B=120650	s/sec=2055.65 tok/sec=98581.54 u/sec=23.32	ppl=6.130791 
[2023-12-17:12:48:38:INFO:sockeye.training:__call__] E=30 B=120700	s/sec=2136.17 tok/sec=98046.80 u/sec=23.38	ppl=6.143848 
[2023-12-17:12:48:40:INFO:sockeye.training:__call__] E=30 B=120750	s/sec=1988.32 tok/sec=99481.41 u/sec=22.72	ppl=6.153911 
[2023-12-17:12:48:42:INFO:sockeye.training:__call__] E=30 B=120800	s/sec=2079.79 tok/sec=100007.21 u/sec=23.09	ppl=6.163001 
[2023-12-17:12:48:44:INFO:sockeye.training:__call__] E=30 B=120850	s/sec=2072.72 tok/sec=100598.17 u/sec=22.77	ppl=6.167255 
[2023-12-17:12:48:46:INFO:sockeye.training:__call__] E=30 B=120900	s/sec=2095.21 tok/sec=98218.85 u/sec=23.55	ppl=6.174272 
[2023-12-17:12:48:48:INFO:sockeye.training:__call__] E=30 B=120950	s/sec=1926.37 tok/sec=102079.16 u/sec=22.76	ppl=6.181559 
[2023-12-17:12:48:51:INFO:sockeye.training:__call__] E=30 B=121000	s/sec=2063.41 tok/sec=99043.53 u/sec=23.41	ppl=6.192053 
[2023-12-17:12:48:53:INFO:sockeye.training:__call__] E=30 B=121050	s/sec=2160.91 tok/sec=97738.60 u/sec=23.74	ppl=6.203075 
[2023-12-17:12:48:55:INFO:sockeye.training:__call__] E=30 B=121100	s/sec=2326.40 tok/sec=92371.08 u/sec=23.26	ppl=6.212951 
[2023-12-17:12:48:57:INFO:sockeye.training:__call__] E=30 B=121150	s/sec=2391.51 tok/sec=94073.88 u/sec=23.84	ppl=6.226667 
[2023-12-17:12:48:59:INFO:sockeye.training:__call__] E=30 B=121200	s/sec=1821.10 tok/sec=101951.86 u/sec=23.13	ppl=6.233826 
[2023-12-17:12:49:01:INFO:sockeye.training:__call__] E=30 B=121250	s/sec=2092.97 tok/sec=101407.64 u/sec=23.07	ppl=6.242899 
[2023-12-17:12:49:04:INFO:sockeye.training:__call__] E=30 B=121300	s/sec=2038.34 tok/sec=105579.23 u/sec=21.59	ppl=6.253408 
[2023-12-17:12:49:06:INFO:sockeye.training:__call__] E=30 B=121350	s/sec=2020.87 tok/sec=98509.34 u/sec=23.09	ppl=6.263427 
[2023-12-17:12:49:08:INFO:sockeye.training:__call__] E=30 B=121400	s/sec=2244.90 tok/sec=99395.90 u/sec=23.08	ppl=6.271587 
[2023-12-17:12:49:10:INFO:sockeye.training:__call__] E=30 B=121450	s/sec=1863.74 tok/sec=101197.29 u/sec=22.53	ppl=6.277202 
[2023-12-17:12:49:12:INFO:sockeye.training:__call__] E=30 B=121500	s/sec=2123.25 tok/sec=99387.29 u/sec=23.24	ppl=6.282804 
[2023-12-17:12:49:14:INFO:sockeye.training:__call__] E=30 B=121550	s/sec=2138.91 tok/sec=99746.93 u/sec=23.05	ppl=6.289426 
[2023-12-17:12:49:17:INFO:sockeye.training:__call__] E=30 B=121600	s/sec=1882.64 tok/sec=101949.77 u/sec=23.03	ppl=6.293663 
[2023-12-17:12:49:19:INFO:sockeye.training:__call__] E=30 B=121650	s/sec=2078.30 tok/sec=99849.19 u/sec=23.62	ppl=6.300416 
[2023-12-17:12:49:21:INFO:sockeye.training:__call__] E=30 B=121700	s/sec=2089.61 tok/sec=99066.41 u/sec=23.53	ppl=6.307333 
[2023-12-17:12:49:23:INFO:sockeye.training:__call__] E=30 B=121750	s/sec=2065.15 tok/sec=98069.86 u/sec=23.60	ppl=6.315204 
[2023-12-17:12:49:25:INFO:sockeye.training:__call__] E=30 B=121800	s/sec=2236.25 tok/sec=96789.41 u/sec=23.89	ppl=6.323144 
[2023-12-17:12:49:27:INFO:sockeye.training:__call__] E=30 B=121850	s/sec=2033.67 tok/sec=101307.27 u/sec=23.07	ppl=6.331324 
[2023-12-17:12:49:29:INFO:sockeye.training:__call__] E=30 B=121900	s/sec=2156.06 tok/sec=97121.56 u/sec=23.81	ppl=6.337424 
[2023-12-17:12:49:32:INFO:sockeye.training:__call__] E=30 B=121950	s/sec=2042.66 tok/sec=98910.88 u/sec=23.25	ppl=6.341862 
[2023-12-17:12:49:34:INFO:sockeye.training:__call__] E=30 B=122000	s/sec=1920.51 tok/sec=99407.73 u/sec=23.13	ppl=6.346966 
[2023-12-17:12:49:36:INFO:sockeye.training:__call__] E=30 B=122050	s/sec=2257.03 tok/sec=96671.73 u/sec=23.55	ppl=6.355797 
[2023-12-17:12:49:38:INFO:sockeye.training:__call__] E=30 B=122100	s/sec=2308.88 tok/sec=98127.56 u/sec=23.73	ppl=6.361303 
[2023-12-17:12:49:40:INFO:sockeye.training:__call__] E=30 B=122150	s/sec=2028.10 tok/sec=98042.40 u/sec=23.56	ppl=6.367209 
[2023-12-17:12:49:42:INFO:sockeye.training:__call__] E=30 B=122200	s/sec=1846.58 tok/sec=101805.65 u/sec=22.76	ppl=6.371346 
[2023-12-17:12:49:44:INFO:sockeye.training:__call__] E=30 B=122250	s/sec=1931.92 tok/sec=102119.99 u/sec=22.36	ppl=6.376792 
[2023-12-17:12:49:47:INFO:sockeye.training:__call__] E=30 B=122300	s/sec=2105.94 tok/sec=99357.76 u/sec=23.67	ppl=6.381665 
[2023-12-17:12:49:49:INFO:sockeye.training:__call__] E=30 B=122350	s/sec=2110.43 tok/sec=99321.44 u/sec=23.43	ppl=6.387562 
[2023-12-17:12:49:51:INFO:sockeye.training:__call__] E=30 B=122400	s/sec=2118.86 tok/sec=101588.21 u/sec=22.91	ppl=6.394936 
[2023-12-17:12:49:53:INFO:sockeye.training:__call__] E=30 B=122450	s/sec=2017.92 tok/sec=102487.97 u/sec=23.14	ppl=6.399089 
[2023-12-17:12:49:55:INFO:sockeye.training:__call__] E=30 B=122500	s/sec=2213.74 tok/sec=95153.00 u/sec=23.77	ppl=6.406726 
[2023-12-17:12:49:57:INFO:sockeye.training:__call__] E=30 B=122550	s/sec=2282.94 tok/sec=92905.88 u/sec=23.86	ppl=6.415493 
[2023-12-17:12:49:59:INFO:sockeye.training:__call__] E=30 B=122600	s/sec=1903.55 tok/sec=101874.71 u/sec=22.92	ppl=6.419243 
[2023-12-17:12:50:02:INFO:sockeye.training:__call__] E=30 B=122650	s/sec=2038.65 tok/sec=100262.25 u/sec=23.51	ppl=6.425460 
[2023-12-17:12:50:04:INFO:sockeye.training:__call__] E=30 B=122700	s/sec=1942.36 tok/sec=103572.99 u/sec=22.69	ppl=6.429802 
[2023-12-17:12:50:06:INFO:sockeye.training:__call__] E=30 B=122750	s/sec=1825.53 tok/sec=101729.37 u/sec=23.00	ppl=6.433258 
[2023-12-17:12:50:08:INFO:sockeye.training:__call__] E=30 B=122800	s/sec=2137.76 tok/sec=101098.11 u/sec=23.20	ppl=6.440060 
[2023-12-17:12:50:10:INFO:sockeye.training:__call__] E=30 B=122850	s/sec=1940.87 tok/sec=101146.32 u/sec=23.02	ppl=6.444306 
[2023-12-17:12:50:12:INFO:sockeye.training:__call__] E=30 B=122900	s/sec=2059.23 tok/sec=99949.25 u/sec=23.36	ppl=6.448955 
[2023-12-17:12:50:15:INFO:sockeye.training:__call__] E=30 B=122950	s/sec=1926.93 tok/sec=103404.92 u/sec=22.55	ppl=6.454321 
[2023-12-17:12:50:17:INFO:sockeye.training:__call__] E=30 B=123000	s/sec=2230.15 tok/sec=96710.30 u/sec=23.54	ppl=6.461241 
[2023-12-17:12:50:19:INFO:sockeye.training:__call__] E=30 B=123050	s/sec=1923.20 tok/sec=101568.22 u/sec=22.81	ppl=6.465424 
[2023-12-17:12:50:21:INFO:sockeye.training:__call__] E=30 B=123100	s/sec=2081.21 tok/sec=99180.79 u/sec=23.35	ppl=6.471900 
[2023-12-17:12:50:23:INFO:sockeye.training:__call__] E=30 B=123150	s/sec=2233.36 tok/sec=98327.44 u/sec=23.26	ppl=6.477397 
[2023-12-17:12:50:25:INFO:sockeye.training:__call__] E=30 B=123200	s/sec=1938.39 tok/sec=100691.92 u/sec=23.34	ppl=6.481510 
[2023-12-17:12:50:28:INFO:sockeye.training:__call__] E=30 B=123250	s/sec=2063.22 tok/sec=99758.62 u/sec=23.57	ppl=6.486472 
[2023-12-17:12:50:30:INFO:sockeye.training:__call__] E=30 B=123300	s/sec=1927.26 tok/sec=102235.66 u/sec=22.68	ppl=6.491170 
[2023-12-17:12:50:32:INFO:sockeye.training:__call__] E=30 B=123350	s/sec=1987.18 tok/sec=99433.46 u/sec=23.26	ppl=6.496745 
[2023-12-17:12:50:34:INFO:sockeye.training:__call__] E=30 B=123400	s/sec=2310.41 tok/sec=97120.69 u/sec=23.71	ppl=6.503223 
[2023-12-17:12:50:36:INFO:sockeye.training:__call__] E=30 B=123450	s/sec=2034.02 tok/sec=98147.37 u/sec=23.63	ppl=6.507832 
[2023-12-17:12:50:38:INFO:sockeye.training:__call__] E=30 B=123500	s/sec=2076.34 tok/sec=101441.72 u/sec=22.77	ppl=6.514132 
[2023-12-17:12:50:40:INFO:sockeye.training:__call__] E=30 B=123550	s/sec=2183.46 tok/sec=99837.03 u/sec=23.53	ppl=6.520098 
[2023-12-17:12:50:43:INFO:sockeye.training:__call__] E=30 B=123600	s/sec=2117.56 tok/sec=99170.50 u/sec=23.85	ppl=6.527101 
[2023-12-17:12:50:45:INFO:sockeye.training:__call__] E=30 B=123650	s/sec=2022.09 tok/sec=97029.95 u/sec=23.62	ppl=6.531397 
[2023-12-17:12:50:47:INFO:sockeye.training:__call__] E=30 B=123700	s/sec=2067.28 tok/sec=96083.42 u/sec=23.41	ppl=6.535777 
[2023-12-17:12:50:49:INFO:sockeye.training:__call__] E=30 B=123750	s/sec=2036.81 tok/sec=99659.38 u/sec=23.10	ppl=6.540494 
[2023-12-17:12:50:51:INFO:sockeye.training:__call__] E=30 B=123800	s/sec=2016.21 tok/sec=102096.38 u/sec=22.71	ppl=6.544403 
[2023-12-17:12:50:53:INFO:sockeye.training:__call__] E=31 B=123850	s/sec=2167.13 tok/sec=90187.46 u/sec=22.61	ppl=6.548731 
[2023-12-17:12:50:56:INFO:sockeye.training:__call__] E=31 B=123900	s/sec=1968.46 tok/sec=102155.87 u/sec=22.78	ppl=6.538768 
[2023-12-17:12:50:58:INFO:sockeye.training:__call__] E=31 B=123950	s/sec=1837.34 tok/sec=104717.90 u/sec=22.30	ppl=6.528820 
[2023-12-17:12:51:00:INFO:sockeye.training:__call__] E=31 B=124000	s/sec=1853.70 tok/sec=103430.80 u/sec=22.63	ppl=6.520493 
[2023-12-17:12:51:00:INFO:sockeye.training:_create_checkpoint] Checkpoint [31]	Updates=124000 Epoch=31 Samples=11050200 Time-cost=172.361 Updates/sec=23.207
[2023-12-17:12:51:00:INFO:sockeye.training:_create_checkpoint] Checkpoint [31]	Train-ppl=6.520493
[2023-12-17:12:51:03:INFO:sockeye.training:_evaluate] Checkpoint [31]	Validation-ppl=32.513231	Validation-bleu=0.162242	Validation-chrf=30.272154	Validation-rouge1=0.158243	Validation-rouge2=0.074517	Validation-rougel=0.151255	Validation-length-ratio-mse=1.291628	Validation-ter=97.276104	Validation-avg-sec-per-sent=0.003340	Validation-decode-walltime=1.670135
[2023-12-17:12:51:03:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 7 checkpoints, best so far: 32.780032
[2023-12-17:12:51:03:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:51:03:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [31]	Learning-rate=0.000162
[2023-12-17:12:51:03:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:51:03:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=7, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:51:03:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00031"
[2023-12-17:12:51:06:INFO:sockeye.training:__call__] E=31 B=124050	s/sec=800.86 tok/sec=38992.41 u/sec=8.97	ppl=5.939929 
[2023-12-17:12:51:08:INFO:sockeye.training:__call__] E=31 B=124100	s/sec=2090.04 tok/sec=95514.17 u/sec=23.92	ppl=5.942169 
[2023-12-17:12:51:10:INFO:sockeye.training:__call__] E=31 B=124150	s/sec=2154.69 tok/sec=98487.62 u/sec=23.10	ppl=5.949938 
[2023-12-17:12:51:12:INFO:sockeye.training:__call__] E=31 B=124200	s/sec=2094.29 tok/sec=101231.51 u/sec=22.96	ppl=5.965230 
[2023-12-17:12:51:14:INFO:sockeye.training:__call__] E=31 B=124250	s/sec=1954.65 tok/sec=101583.25 u/sec=22.96	ppl=5.975937 
[2023-12-17:12:51:16:INFO:sockeye.training:__call__] E=31 B=124300	s/sec=2250.57 tok/sec=97496.98 u/sec=23.64	ppl=5.978624 
[2023-12-17:12:51:18:INFO:sockeye.training:__call__] E=31 B=124350	s/sec=2133.78 tok/sec=97776.73 u/sec=23.56	ppl=5.989525 
[2023-12-17:12:51:21:INFO:sockeye.training:__call__] E=31 B=124400	s/sec=2097.76 tok/sec=97815.33 u/sec=23.41	ppl=6.000140 
[2023-12-17:12:51:23:INFO:sockeye.training:__call__] E=31 B=124450	s/sec=2018.43 tok/sec=100387.42 u/sec=23.19	ppl=6.003626 
[2023-12-17:12:51:25:INFO:sockeye.training:__call__] E=31 B=124500	s/sec=2099.65 tok/sec=96675.81 u/sec=23.95	ppl=6.005359 
[2023-12-17:12:51:27:INFO:sockeye.training:__call__] E=31 B=124550	s/sec=2373.10 tok/sec=95905.38 u/sec=23.96	ppl=6.023998 
[2023-12-17:12:51:29:INFO:sockeye.training:__call__] E=31 B=124600	s/sec=2091.27 tok/sec=97760.29 u/sec=23.81	ppl=6.039225 
[2023-12-17:12:51:31:INFO:sockeye.training:__call__] E=31 B=124650	s/sec=2035.72 tok/sec=108711.72 u/sec=21.86	ppl=6.046731 
[2023-12-17:12:51:33:INFO:sockeye.training:__call__] E=31 B=124700	s/sec=2165.73 tok/sec=96820.97 u/sec=23.42	ppl=6.056066 
[2023-12-17:12:51:36:INFO:sockeye.training:__call__] E=31 B=124750	s/sec=2043.61 tok/sec=98003.68 u/sec=23.31	ppl=6.065608 
[2023-12-17:12:51:38:INFO:sockeye.training:__call__] E=31 B=124800	s/sec=2078.69 tok/sec=98226.78 u/sec=23.75	ppl=6.073669 
[2023-12-17:12:51:40:INFO:sockeye.training:__call__] E=31 B=124850	s/sec=2036.99 tok/sec=100154.65 u/sec=23.23	ppl=6.078134 
[2023-12-17:12:51:42:INFO:sockeye.training:__call__] E=31 B=124900	s/sec=2228.59 tok/sec=98582.15 u/sec=23.41	ppl=6.089361 
[2023-12-17:12:51:44:INFO:sockeye.training:__call__] E=31 B=124950	s/sec=2227.85 tok/sec=96843.36 u/sec=23.68	ppl=6.100649 
[2023-12-17:12:51:46:INFO:sockeye.training:__call__] E=31 B=125000	s/sec=2099.62 tok/sec=98462.69 u/sec=23.23	ppl=6.108906 
[2023-12-17:12:51:48:INFO:sockeye.training:__call__] E=31 B=125050	s/sec=2082.55 tok/sec=98603.49 u/sec=23.08	ppl=6.112743 
[2023-12-17:12:51:51:INFO:sockeye.training:__call__] E=31 B=125100	s/sec=2209.86 tok/sec=96765.53 u/sec=23.61	ppl=6.124427 
[2023-12-17:12:51:53:INFO:sockeye.training:__call__] E=31 B=125150	s/sec=2089.63 tok/sec=97435.64 u/sec=23.57	ppl=6.129604 
[2023-12-17:12:51:55:INFO:sockeye.training:__call__] E=31 B=125200	s/sec=2146.14 tok/sec=98871.67 u/sec=23.29	ppl=6.135539 
[2023-12-17:12:51:57:INFO:sockeye.training:__call__] E=31 B=125250	s/sec=2070.61 tok/sec=102022.05 u/sec=23.11	ppl=6.142744 
[2023-12-17:12:51:59:INFO:sockeye.training:__call__] E=31 B=125300	s/sec=2126.49 tok/sec=99156.75 u/sec=23.24	ppl=6.151482 
[2023-12-17:12:52:01:INFO:sockeye.training:__call__] E=31 B=125350	s/sec=2113.17 tok/sec=95666.13 u/sec=23.58	ppl=6.159249 
[2023-12-17:12:52:03:INFO:sockeye.training:__call__] E=31 B=125400	s/sec=1923.43 tok/sec=104433.23 u/sec=22.47	ppl=6.165738 
[2023-12-17:12:52:06:INFO:sockeye.training:__call__] E=31 B=125450	s/sec=1936.68 tok/sec=101427.66 u/sec=22.97	ppl=6.170181 
[2023-12-17:12:52:08:INFO:sockeye.training:__call__] E=31 B=125500	s/sec=1972.76 tok/sec=99852.73 u/sec=23.58	ppl=6.177052 
[2023-12-17:12:52:10:INFO:sockeye.training:__call__] E=31 B=125550	s/sec=2138.28 tok/sec=99263.91 u/sec=23.32	ppl=6.184903 
[2023-12-17:12:52:12:INFO:sockeye.training:__call__] E=31 B=125600	s/sec=2017.88 tok/sec=99445.69 u/sec=22.97	ppl=6.192311 
[2023-12-17:12:52:14:INFO:sockeye.training:__call__] E=31 B=125650	s/sec=2032.89 tok/sec=96810.10 u/sec=24.02	ppl=6.197351 
[2023-12-17:12:52:16:INFO:sockeye.training:__call__] E=31 B=125700	s/sec=1838.70 tok/sec=102150.12 u/sec=22.80	ppl=6.202992 
[2023-12-17:12:52:19:INFO:sockeye.training:__call__] E=31 B=125750	s/sec=1808.38 tok/sec=104697.24 u/sec=22.70	ppl=6.205059 
[2023-12-17:12:52:21:INFO:sockeye.training:__call__] E=31 B=125800	s/sec=2259.75 tok/sec=98240.94 u/sec=23.50	ppl=6.213555 
[2023-12-17:12:52:23:INFO:sockeye.training:__call__] E=31 B=125850	s/sec=2243.99 tok/sec=95236.65 u/sec=23.77	ppl=6.220610 
[2023-12-17:12:52:25:INFO:sockeye.training:__call__] E=31 B=125900	s/sec=1973.23 tok/sec=100552.67 u/sec=23.27	ppl=6.225278 
[2023-12-17:12:52:27:INFO:sockeye.training:__call__] E=31 B=125950	s/sec=2089.76 tok/sec=98832.69 u/sec=23.53	ppl=6.233761 
[2023-12-17:12:52:29:INFO:sockeye.training:__call__] E=31 B=126000	s/sec=2250.00 tok/sec=98556.11 u/sec=23.13	ppl=6.241876 
[2023-12-17:12:52:31:INFO:sockeye.training:__call__] E=31 B=126050	s/sec=2205.31 tok/sec=95132.23 u/sec=24.35	ppl=6.247478 
[2023-12-17:12:52:33:INFO:sockeye.training:__call__] E=31 B=126100	s/sec=2157.39 tok/sec=98861.06 u/sec=23.21	ppl=6.253400 
[2023-12-17:12:52:36:INFO:sockeye.training:__call__] E=31 B=126150	s/sec=2009.19 tok/sec=102805.63 u/sec=23.13	ppl=6.257214 
[2023-12-17:12:52:38:INFO:sockeye.training:__call__] E=31 B=126200	s/sec=2225.62 tok/sec=99398.84 u/sec=23.22	ppl=6.263105 
[2023-12-17:12:52:40:INFO:sockeye.training:__call__] E=31 B=126250	s/sec=2010.06 tok/sec=101408.11 u/sec=23.18	ppl=6.269051 
[2023-12-17:12:52:42:INFO:sockeye.training:__call__] E=31 B=126300	s/sec=2021.89 tok/sec=100029.71 u/sec=23.27	ppl=6.274661 
[2023-12-17:12:52:44:INFO:sockeye.training:__call__] E=31 B=126350	s/sec=1992.87 tok/sec=99672.56 u/sec=22.73	ppl=6.280313 
[2023-12-17:12:52:46:INFO:sockeye.training:__call__] E=31 B=126400	s/sec=2081.29 tok/sec=100170.39 u/sec=23.31	ppl=6.286879 
[2023-12-17:12:52:49:INFO:sockeye.training:__call__] E=31 B=126450	s/sec=2127.15 tok/sec=97344.37 u/sec=23.53	ppl=6.293924 
[2023-12-17:12:52:51:INFO:sockeye.training:__call__] E=31 B=126500	s/sec=2165.99 tok/sec=100172.43 u/sec=22.98	ppl=6.299711 
[2023-12-17:12:52:53:INFO:sockeye.training:__call__] E=31 B=126550	s/sec=2053.87 tok/sec=99485.03 u/sec=23.42	ppl=6.304812 
[2023-12-17:12:52:55:INFO:sockeye.training:__call__] E=31 B=126600	s/sec=2108.23 tok/sec=100840.00 u/sec=23.12	ppl=6.310980 
[2023-12-17:12:52:57:INFO:sockeye.training:__call__] E=31 B=126650	s/sec=2047.48 tok/sec=97269.59 u/sec=22.53	ppl=6.315139 
[2023-12-17:12:52:59:INFO:sockeye.training:__call__] E=31 B=126700	s/sec=1986.88 tok/sec=98061.81 u/sec=22.14	ppl=6.319921 
[2023-12-17:12:53:02:INFO:sockeye.training:__call__] E=31 B=126750	s/sec=1881.83 tok/sec=101122.22 u/sec=22.49	ppl=6.324184 
[2023-12-17:12:53:04:INFO:sockeye.training:__call__] E=31 B=126800	s/sec=1908.95 tok/sec=102296.45 u/sec=22.77	ppl=6.327793 
[2023-12-17:12:53:06:INFO:sockeye.training:__call__] E=31 B=126850	s/sec=2176.96 tok/sec=95923.69 u/sec=23.83	ppl=6.334134 
[2023-12-17:12:53:08:INFO:sockeye.training:__call__] E=31 B=126900	s/sec=2040.02 tok/sec=100789.02 u/sec=23.10	ppl=6.339126 
[2023-12-17:12:53:10:INFO:sockeye.training:__call__] E=31 B=126950	s/sec=2024.17 tok/sec=100920.05 u/sec=23.13	ppl=6.344355 
[2023-12-17:12:53:12:INFO:sockeye.training:__call__] E=31 B=127000	s/sec=2008.30 tok/sec=101212.44 u/sec=23.07	ppl=6.349428 
[2023-12-17:12:53:15:INFO:sockeye.training:__call__] E=31 B=127050	s/sec=2010.19 tok/sec=101523.70 u/sec=22.48	ppl=6.356045 
[2023-12-17:12:53:17:INFO:sockeye.training:__call__] E=31 B=127100	s/sec=2106.30 tok/sec=94530.25 u/sec=22.62	ppl=6.360590 
[2023-12-17:12:53:19:INFO:sockeye.training:__call__] E=31 B=127150	s/sec=2171.14 tok/sec=92898.68 u/sec=22.84	ppl=6.366728 
[2023-12-17:12:53:21:INFO:sockeye.training:__call__] E=31 B=127200	s/sec=2086.91 tok/sec=98417.59 u/sec=23.63	ppl=6.373367 
[2023-12-17:12:53:23:INFO:sockeye.training:__call__] E=31 B=127250	s/sec=2000.90 tok/sec=98055.09 u/sec=22.45	ppl=6.378579 
[2023-12-17:12:53:26:INFO:sockeye.training:__call__] E=31 B=127300	s/sec=2113.63 tok/sec=95205.71 u/sec=23.14	ppl=6.385454 
[2023-12-17:12:53:28:INFO:sockeye.training:__call__] E=31 B=127350	s/sec=1743.23 tok/sec=104957.84 u/sec=21.88	ppl=6.388661 
[2023-12-17:12:53:30:INFO:sockeye.training:__call__] E=31 B=127400	s/sec=2175.91 tok/sec=95620.03 u/sec=23.45	ppl=6.394994 
[2023-12-17:12:53:32:INFO:sockeye.training:__call__] E=31 B=127450	s/sec=2122.35 tok/sec=95165.24 u/sec=23.39	ppl=6.400450 
[2023-12-17:12:53:34:INFO:sockeye.training:__call__] E=31 B=127500	s/sec=1987.79 tok/sec=99745.32 u/sec=23.18	ppl=6.405083 
[2023-12-17:12:53:36:INFO:sockeye.training:__call__] E=31 B=127550	s/sec=2307.85 tok/sec=95257.08 u/sec=24.20	ppl=6.411771 
[2023-12-17:12:53:38:INFO:sockeye.training:__call__] E=31 B=127600	s/sec=2202.18 tok/sec=96451.55 u/sec=23.94	ppl=6.417297 
[2023-12-17:12:53:41:INFO:sockeye.training:__call__] E=31 B=127650	s/sec=1941.24 tok/sec=100328.87 u/sec=23.47	ppl=6.420641 
[2023-12-17:12:53:43:INFO:sockeye.training:__call__] E=31 B=127700	s/sec=1986.90 tok/sec=97304.60 u/sec=22.21	ppl=6.423993 
[2023-12-17:12:53:45:INFO:sockeye.training:__call__] E=31 B=127750	s/sec=1846.01 tok/sec=99362.14 u/sec=22.40	ppl=6.428173 
[2023-12-17:12:53:47:INFO:sockeye.training:__call__] E=31 B=127800	s/sec=2010.97 tok/sec=97145.39 u/sec=23.02	ppl=6.434289 
[2023-12-17:12:53:50:INFO:sockeye.training:__call__] E=32 B=127850	s/sec=1858.11 tok/sec=89298.35 u/sec=21.27	ppl=6.437334 
[2023-12-17:12:53:52:INFO:sockeye.training:__call__] E=32 B=127900	s/sec=2122.66 tok/sec=96522.04 u/sec=23.82	ppl=6.429154 
[2023-12-17:12:53:54:INFO:sockeye.training:__call__] E=32 B=127950	s/sec=1730.47 tok/sec=104192.41 u/sec=22.77	ppl=6.419698 
[2023-12-17:12:53:56:INFO:sockeye.training:__call__] E=32 B=128000	s/sec=2087.93 tok/sec=99322.47 u/sec=23.39	ppl=6.412820 
[2023-12-17:12:53:56:INFO:sockeye.training:_create_checkpoint] Checkpoint [32]	Updates=128000 Epoch=32 Samples=11406968 Time-cost=172.644 Updates/sec=23.169
[2023-12-17:12:53:56:INFO:sockeye.training:_create_checkpoint] Checkpoint [32]	Train-ppl=6.412820
[2023-12-17:12:53:59:INFO:sockeye.training:_evaluate] Checkpoint [32]	Validation-ppl=33.819907	Validation-bleu=0.194085	Validation-chrf=32.498671	Validation-rouge1=0.167656	Validation-rouge2=0.085043	Validation-rougel=0.159627	Validation-length-ratio-mse=1.309333	Validation-ter=95.996698	Validation-avg-sec-per-sent=0.003667	Validation-decode-walltime=1.833360
[2023-12-17:12:53:59:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 8 checkpoints, best so far: 32.780032
[2023-12-17:12:53:59:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:53:59:INFO:sockeye.lr_scheduler:new_evaluation_result] 8 checkpoints since improvement or rate scaling, lowering learning rate: 1.62e-04 -> 1.46e-04
[2023-12-17:12:53:59:INFO:sockeye.training:_adjust_learning_rate] Loading model parameters and optimizer states from best checkpoint: 24
[2023-12-17:12:53:59:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best" to "cuda:0"
[2023-12-17:12:53:59:INFO:sockeye.training:_load_optimizer_state] Loaded optimizer state from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:12:53:59:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [32]	Learning-rate=0.000162
[2023-12-17:12:54:00:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:54:00:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:54:00:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00032"
[2023-12-17:12:54:02:INFO:sockeye.training:__call__] E=32 B=128050	s/sec=785.34 tok/sec=32926.20 u/sec=8.26	ppl=6.996054 
[2023-12-17:12:54:04:INFO:sockeye.training:__call__] E=32 B=128100	s/sec=1914.14 tok/sec=102693.12 u/sec=22.53	ppl=6.887231 
[2023-12-17:12:54:06:INFO:sockeye.training:__call__] E=32 B=128150	s/sec=2100.78 tok/sec=96979.56 u/sec=23.36	ppl=6.882953 
[2023-12-17:12:54:09:INFO:sockeye.training:__call__] E=32 B=128200	s/sec=1808.31 tok/sec=100543.48 u/sec=23.50	ppl=6.883384 
[2023-12-17:12:54:11:INFO:sockeye.training:__call__] E=32 B=128250	s/sec=1951.58 tok/sec=100176.38 u/sec=23.19	ppl=6.887496 
[2023-12-17:12:54:13:INFO:sockeye.training:__call__] E=32 B=128300	s/sec=1987.26 tok/sec=100737.69 u/sec=23.35	ppl=6.891119 
[2023-12-17:12:54:15:INFO:sockeye.training:__call__] E=32 B=128350	s/sec=2124.66 tok/sec=99525.43 u/sec=23.42	ppl=6.908195 
[2023-12-17:12:54:17:INFO:sockeye.training:__call__] E=32 B=128400	s/sec=2066.99 tok/sec=101231.42 u/sec=22.82	ppl=6.920443 
[2023-12-17:12:54:19:INFO:sockeye.training:__call__] E=32 B=128450	s/sec=1870.30 tok/sec=100487.74 u/sec=23.38	ppl=6.914404 
[2023-12-17:12:54:22:INFO:sockeye.training:__call__] E=32 B=128500	s/sec=2042.53 tok/sec=100477.74 u/sec=23.21	ppl=6.922321 
[2023-12-17:12:54:24:INFO:sockeye.training:__call__] E=32 B=128550	s/sec=2427.35 tok/sec=94354.94 u/sec=24.04	ppl=6.949002 
[2023-12-17:12:54:26:INFO:sockeye.training:__call__] E=32 B=128600	s/sec=2206.64 tok/sec=99955.40 u/sec=23.06	ppl=6.962220 
[2023-12-17:12:54:28:INFO:sockeye.training:__call__] E=32 B=128650	s/sec=2024.29 tok/sec=98176.36 u/sec=23.21	ppl=6.974773 
[2023-12-17:12:54:30:INFO:sockeye.training:__call__] E=32 B=128700	s/sec=2112.15 tok/sec=100009.97 u/sec=23.32	ppl=6.984411 
[2023-12-17:12:54:32:INFO:sockeye.training:__call__] E=32 B=128750	s/sec=1952.48 tok/sec=104450.46 u/sec=22.60	ppl=6.985552 
[2023-12-17:12:54:34:INFO:sockeye.training:__call__] E=32 B=128800	s/sec=2264.15 tok/sec=96998.13 u/sec=23.70	ppl=7.003218 
[2023-12-17:12:54:37:INFO:sockeye.training:__call__] E=32 B=128850	s/sec=2122.99 tok/sec=98799.32 u/sec=23.32	ppl=7.014239 
[2023-12-17:12:54:39:INFO:sockeye.training:__call__] E=32 B=128900	s/sec=2093.62 tok/sec=95912.50 u/sec=23.24	ppl=7.024351 
[2023-12-17:12:54:41:INFO:sockeye.training:__call__] E=32 B=128950	s/sec=2023.19 tok/sec=100870.76 u/sec=23.12	ppl=7.030060 
[2023-12-17:12:54:43:INFO:sockeye.training:__call__] E=32 B=129000	s/sec=1950.87 tok/sec=101005.29 u/sec=22.92	ppl=7.033179 
[2023-12-17:12:54:45:INFO:sockeye.training:__call__] E=32 B=129050	s/sec=1758.11 tok/sec=105472.07 u/sec=22.29	ppl=7.042317 
[2023-12-17:12:54:47:INFO:sockeye.training:__call__] E=32 B=129100	s/sec=2085.70 tok/sec=100528.50 u/sec=23.15	ppl=7.051914 
[2023-12-17:12:54:50:INFO:sockeye.training:__call__] E=32 B=129150	s/sec=2044.82 tok/sec=98332.17 u/sec=23.54	ppl=7.061000 
[2023-12-17:12:54:52:INFO:sockeye.training:__call__] E=32 B=129200	s/sec=1939.40 tok/sec=100162.19 u/sec=22.83	ppl=7.068981 
[2023-12-17:12:54:54:INFO:sockeye.training:__call__] E=32 B=129250	s/sec=2329.39 tok/sec=91620.08 u/sec=24.30	ppl=7.081442 
[2023-12-17:12:54:56:INFO:sockeye.training:__call__] E=32 B=129300	s/sec=2240.53 tok/sec=99555.98 u/sec=23.38	ppl=7.090223 
[2023-12-17:12:54:58:INFO:sockeye.training:__call__] E=32 B=129350	s/sec=2175.44 tok/sec=97838.50 u/sec=23.48	ppl=7.103118 
[2023-12-17:12:55:00:INFO:sockeye.training:__call__] E=32 B=129400	s/sec=2339.22 tok/sec=94401.07 u/sec=24.09	ppl=7.120479 
[2023-12-17:12:55:02:INFO:sockeye.training:__call__] E=32 B=129450	s/sec=2121.82 tok/sec=95966.91 u/sec=23.68	ppl=7.136084 
[2023-12-17:12:55:04:INFO:sockeye.training:__call__] E=32 B=129500	s/sec=2150.13 tok/sec=96361.84 u/sec=23.45	ppl=7.145890 
[2023-12-17:12:55:07:INFO:sockeye.training:__call__] E=32 B=129550	s/sec=2301.11 tok/sec=97040.74 u/sec=23.23	ppl=7.154214 
[2023-12-17:12:55:09:INFO:sockeye.training:__call__] E=32 B=129600	s/sec=2107.57 tok/sec=103299.66 u/sec=22.56	ppl=7.163111 
[2023-12-17:12:55:11:INFO:sockeye.training:__call__] E=32 B=129650	s/sec=2056.21 tok/sec=101651.34 u/sec=23.07	ppl=7.168136 
[2023-12-17:12:55:13:INFO:sockeye.training:__call__] E=32 B=129700	s/sec=2130.68 tok/sec=100292.06 u/sec=23.44	ppl=7.175560 
[2023-12-17:12:55:15:INFO:sockeye.training:__call__] E=32 B=129750	s/sec=1986.81 tok/sec=102304.25 u/sec=22.87	ppl=7.184171 
[2023-12-17:12:55:17:INFO:sockeye.training:__call__] E=32 B=129800	s/sec=1919.34 tok/sec=99221.80 u/sec=23.38	ppl=7.190470 
[2023-12-17:12:55:20:INFO:sockeye.training:__call__] E=32 B=129850	s/sec=2005.75 tok/sec=103052.45 u/sec=22.92	ppl=7.196196 
[2023-12-17:12:55:22:INFO:sockeye.training:__call__] E=32 B=129900	s/sec=2192.25 tok/sec=97269.78 u/sec=23.46	ppl=7.204367 
[2023-12-17:12:55:24:INFO:sockeye.training:__call__] E=32 B=129950	s/sec=1927.84 tok/sec=103729.37 u/sec=22.48	ppl=7.211709 
[2023-12-17:12:55:26:INFO:sockeye.training:__call__] E=32 B=130000	s/sec=1790.78 tok/sec=101638.69 u/sec=22.52	ppl=7.219341 
[2023-12-17:12:55:28:INFO:sockeye.training:__call__] E=32 B=130050	s/sec=2126.87 tok/sec=97880.76 u/sec=23.32	ppl=7.227915 
[2023-12-17:12:55:30:INFO:sockeye.training:__call__] E=32 B=130100	s/sec=2033.69 tok/sec=101190.09 u/sec=23.07	ppl=7.235880 
[2023-12-17:12:55:33:INFO:sockeye.training:__call__] E=32 B=130150	s/sec=2256.44 tok/sec=96938.77 u/sec=23.94	ppl=7.245216 
[2023-12-17:12:55:35:INFO:sockeye.training:__call__] E=32 B=130200	s/sec=2029.53 tok/sec=99224.81 u/sec=23.53	ppl=7.252532 
[2023-12-17:12:55:37:INFO:sockeye.training:__call__] E=32 B=130250	s/sec=2112.67 tok/sec=96552.97 u/sec=23.71	ppl=7.262039 
[2023-12-17:12:55:39:INFO:sockeye.training:__call__] E=32 B=130300	s/sec=1919.14 tok/sec=101377.69 u/sec=22.89	ppl=7.265777 
[2023-12-17:12:55:41:INFO:sockeye.training:__call__] E=32 B=130350	s/sec=2005.28 tok/sec=103576.90 u/sec=22.70	ppl=7.271218 
[2023-12-17:12:55:43:INFO:sockeye.training:__call__] E=32 B=130400	s/sec=2235.00 tok/sec=95779.35 u/sec=23.96	ppl=7.280607 
[2023-12-17:12:55:46:INFO:sockeye.training:__call__] E=32 B=130450	s/sec=1759.89 tok/sec=110673.01 u/sec=21.57	ppl=7.284384 
[2023-12-17:12:55:48:INFO:sockeye.training:__call__] E=32 B=130500	s/sec=2094.98 tok/sec=99024.71 u/sec=23.51	ppl=7.290495 
[2023-12-17:12:55:50:INFO:sockeye.training:__call__] E=32 B=130550	s/sec=1995.04 tok/sec=100309.40 u/sec=22.92	ppl=7.295493 
[2023-12-17:12:55:52:INFO:sockeye.training:__call__] E=32 B=130600	s/sec=2193.99 tok/sec=96851.83 u/sec=23.52	ppl=7.303905 
[2023-12-17:12:55:54:INFO:sockeye.training:__call__] E=32 B=130650	s/sec=2046.60 tok/sec=98028.09 u/sec=23.30	ppl=7.311559 
[2023-12-17:12:55:56:INFO:sockeye.training:__call__] E=32 B=130700	s/sec=2150.68 tok/sec=98931.15 u/sec=23.18	ppl=7.319783 
[2023-12-17:12:55:58:INFO:sockeye.training:__call__] E=32 B=130750	s/sec=1988.30 tok/sec=101421.88 u/sec=23.14	ppl=7.326070 
[2023-12-17:12:56:01:INFO:sockeye.training:__call__] E=32 B=130800	s/sec=2170.21 tok/sec=97129.03 u/sec=23.51	ppl=7.335326 
[2023-12-17:12:56:03:INFO:sockeye.training:__call__] E=32 B=130850	s/sec=2247.29 tok/sec=94161.85 u/sec=24.17	ppl=7.344741 
[2023-12-17:12:56:05:INFO:sockeye.training:__call__] E=32 B=130900	s/sec=1960.61 tok/sec=100575.06 u/sec=23.39	ppl=7.350552 
[2023-12-17:12:56:07:INFO:sockeye.training:__call__] E=32 B=130950	s/sec=1978.38 tok/sec=102300.44 u/sec=22.48	ppl=7.357275 
[2023-12-17:12:56:09:INFO:sockeye.training:__call__] E=32 B=131000	s/sec=2317.09 tok/sec=97265.48 u/sec=23.40	ppl=7.366988 
[2023-12-17:12:56:11:INFO:sockeye.training:__call__] E=32 B=131050	s/sec=2242.32 tok/sec=98558.31 u/sec=23.16	ppl=7.375553 
[2023-12-17:12:56:13:INFO:sockeye.training:__call__] E=32 B=131100	s/sec=2115.97 tok/sec=95832.94 u/sec=23.70	ppl=7.381509 
[2023-12-17:12:56:16:INFO:sockeye.training:__call__] E=32 B=131150	s/sec=1951.15 tok/sec=99430.68 u/sec=23.14	ppl=7.385404 
[2023-12-17:12:56:18:INFO:sockeye.training:__call__] E=32 B=131200	s/sec=2056.39 tok/sec=99735.03 u/sec=22.95	ppl=7.392294 
[2023-12-17:12:56:20:INFO:sockeye.training:__call__] E=32 B=131250	s/sec=1992.02 tok/sec=101124.10 u/sec=22.89	ppl=7.396039 
[2023-12-17:12:56:22:INFO:sockeye.training:__call__] E=32 B=131300	s/sec=2151.55 tok/sec=97916.07 u/sec=23.39	ppl=7.404788 
[2023-12-17:12:56:24:INFO:sockeye.training:__call__] E=32 B=131350	s/sec=1963.38 tok/sec=99674.78 u/sec=23.07	ppl=7.410381 
[2023-12-17:12:56:26:INFO:sockeye.training:__call__] E=32 B=131400	s/sec=2128.89 tok/sec=97966.66 u/sec=23.72	ppl=7.414425 
[2023-12-17:12:56:28:INFO:sockeye.training:__call__] E=32 B=131450	s/sec=2165.06 tok/sec=98835.90 u/sec=23.66	ppl=7.420313 
[2023-12-17:12:56:31:INFO:sockeye.training:__call__] E=32 B=131500	s/sec=2075.47 tok/sec=100416.15 u/sec=22.96	ppl=7.426189 
[2023-12-17:12:56:33:INFO:sockeye.training:__call__] E=32 B=131550	s/sec=2169.64 tok/sec=96058.82 u/sec=23.83	ppl=7.433141 
[2023-12-17:12:56:35:INFO:sockeye.training:__call__] E=32 B=131600	s/sec=2080.60 tok/sec=100460.07 u/sec=23.10	ppl=7.440000 
[2023-12-17:12:56:37:INFO:sockeye.training:__call__] E=32 B=131650	s/sec=2123.07 tok/sec=97402.53 u/sec=23.78	ppl=7.448191 
[2023-12-17:12:56:39:INFO:sockeye.training:__call__] E=32 B=131700	s/sec=2029.90 tok/sec=102216.34 u/sec=22.78	ppl=7.453323 
[2023-12-17:12:56:41:INFO:sockeye.training:__call__] E=32 B=131750	s/sec=1918.58 tok/sec=101998.69 u/sec=23.10	ppl=7.458733 
[2023-12-17:12:56:44:INFO:sockeye.training:__call__] E=32 B=131800	s/sec=2128.86 tok/sec=98779.20 u/sec=23.14	ppl=7.465096 
[2023-12-17:12:56:46:INFO:sockeye.training:__call__] E=33 B=131850	s/sec=2007.98 tok/sec=91761.34 u/sec=22.02	ppl=7.465968 
[2023-12-17:12:56:48:INFO:sockeye.training:__call__] E=33 B=131900	s/sec=1986.20 tok/sec=99184.03 u/sec=23.12	ppl=7.454293 
[2023-12-17:12:56:50:INFO:sockeye.training:__call__] E=33 B=131950	s/sec=1953.48 tok/sec=103266.85 u/sec=22.91	ppl=7.442066 
[2023-12-17:12:56:52:INFO:sockeye.training:__call__] E=33 B=132000	s/sec=2141.52 tok/sec=97010.74 u/sec=23.90	ppl=7.431422 
[2023-12-17:12:56:52:INFO:sockeye.training:_create_checkpoint] Checkpoint [33]	Updates=132000 Epoch=33 Samples=11763832 Time-cost=172.248 Updates/sec=23.222
[2023-12-17:12:56:52:INFO:sockeye.training:_create_checkpoint] Checkpoint [33]	Train-ppl=7.431422
[2023-12-17:12:56:55:INFO:sockeye.training:_evaluate] Checkpoint [33]	Validation-ppl=29.339310	Validation-bleu=0.173656	Validation-chrf=31.774343	Validation-rouge1=0.169756	Validation-rouge2=0.079812	Validation-rougel=0.161685	Validation-length-ratio-mse=1.274281	Validation-ter=95.583987	Validation-avg-sec-per-sent=0.003300	Validation-decode-walltime=1.650136
[2023-12-17:12:56:55:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 9 checkpoints, best so far: 32.780032
[2023-12-17:12:56:55:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:56:55:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [33]	Learning-rate=0.000146
[2023-12-17:12:56:55:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:56:55:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:56:56:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00033"
[2023-12-17:12:56:58:INFO:sockeye.training:__call__] E=33 B=132050	s/sec=771.13 tok/sec=38042.63 u/sec=8.83	ppl=6.634690 
[2023-12-17:12:57:00:INFO:sockeye.training:__call__] E=33 B=132100	s/sec=2128.83 tok/sec=101117.38 u/sec=23.14	ppl=6.737788 
[2023-12-17:12:57:02:INFO:sockeye.training:__call__] E=33 B=132150	s/sec=2028.64 tok/sec=101088.35 u/sec=22.85	ppl=6.724889 
[2023-12-17:12:57:04:INFO:sockeye.training:__call__] E=33 B=132200	s/sec=2148.41 tok/sec=99769.33 u/sec=23.19	ppl=6.741750 
[2023-12-17:12:57:07:INFO:sockeye.training:__call__] E=33 B=132250	s/sec=2162.79 tok/sec=98857.63 u/sec=23.19	ppl=6.739020 
[2023-12-17:12:57:09:INFO:sockeye.training:__call__] E=33 B=132300	s/sec=2056.35 tok/sec=100325.93 u/sec=23.45	ppl=6.763191 
[2023-12-17:12:57:11:INFO:sockeye.training:__call__] E=33 B=132350	s/sec=1966.24 tok/sec=102883.17 u/sec=22.67	ppl=6.748824 
[2023-12-17:12:57:13:INFO:sockeye.training:__call__] E=33 B=132400	s/sec=2132.40 tok/sec=102296.20 u/sec=23.10	ppl=6.763359 
[2023-12-17:12:57:15:INFO:sockeye.training:__call__] E=33 B=132450	s/sec=1979.59 tok/sec=99120.66 u/sec=23.21	ppl=6.768259 
[2023-12-17:12:57:17:INFO:sockeye.training:__call__] E=33 B=132500	s/sec=1992.51 tok/sec=100211.95 u/sec=23.19	ppl=6.774362 
[2023-12-17:12:57:20:INFO:sockeye.training:__call__] E=33 B=132550	s/sec=2124.80 tok/sec=98041.97 u/sec=23.55	ppl=6.790502 
[2023-12-17:12:57:22:INFO:sockeye.training:__call__] E=33 B=132600	s/sec=2352.09 tok/sec=95852.30 u/sec=23.83	ppl=6.811955 
[2023-12-17:12:57:24:INFO:sockeye.training:__call__] E=33 B=132650	s/sec=2006.59 tok/sec=100500.29 u/sec=23.18	ppl=6.817039 
[2023-12-17:12:57:26:INFO:sockeye.training:__call__] E=33 B=132700	s/sec=2260.19 tok/sec=96363.33 u/sec=23.98	ppl=6.828204 
[2023-12-17:12:57:28:INFO:sockeye.training:__call__] E=33 B=132750	s/sec=2358.83 tok/sec=94898.30 u/sec=23.66	ppl=6.843222 
[2023-12-17:12:57:30:INFO:sockeye.training:__call__] E=33 B=132800	s/sec=2073.78 tok/sec=96766.03 u/sec=24.09	ppl=6.850244 
[2023-12-17:12:57:32:INFO:sockeye.training:__call__] E=33 B=132850	s/sec=2179.30 tok/sec=97674.89 u/sec=23.65	ppl=6.858884 
[2023-12-17:12:57:34:INFO:sockeye.training:__call__] E=33 B=132900	s/sec=2111.96 tok/sec=100814.92 u/sec=23.00	ppl=6.865022 
[2023-12-17:12:57:36:INFO:sockeye.training:__call__] E=33 B=132950	s/sec=2308.50 tok/sec=93732.01 u/sec=24.17	ppl=6.880340 
[2023-12-17:12:57:39:INFO:sockeye.training:__call__] E=33 B=133000	s/sec=1801.08 tok/sec=105376.04 u/sec=22.47	ppl=6.885011 
[2023-12-17:12:57:41:INFO:sockeye.training:__call__] E=33 B=133050	s/sec=1923.78 tok/sec=102212.36 u/sec=22.82	ppl=6.893814 
[2023-12-17:12:57:43:INFO:sockeye.training:__call__] E=33 B=133100	s/sec=1978.80 tok/sec=102513.76 u/sec=23.07	ppl=6.898028 
[2023-12-17:12:57:45:INFO:sockeye.training:__call__] E=33 B=133150	s/sec=2308.87 tok/sec=93098.61 u/sec=24.21	ppl=6.909007 
[2023-12-17:12:57:47:INFO:sockeye.training:__call__] E=33 B=133200	s/sec=2159.97 tok/sec=98603.54 u/sec=23.60	ppl=6.917596 
[2023-12-17:12:57:49:INFO:sockeye.training:__call__] E=33 B=133250	s/sec=1993.57 tok/sec=106897.01 u/sec=21.67	ppl=6.926915 
[2023-12-17:12:57:52:INFO:sockeye.training:__call__] E=33 B=133300	s/sec=1801.89 tok/sec=100701.65 u/sec=22.80	ppl=6.932545 
[2023-12-17:12:57:54:INFO:sockeye.training:__call__] E=33 B=133350	s/sec=2214.20 tok/sec=97000.05 u/sec=23.70	ppl=6.940896 
[2023-12-17:12:57:56:INFO:sockeye.training:__call__] E=33 B=133400	s/sec=2098.00 tok/sec=99911.70 u/sec=22.92	ppl=6.946162 
[2023-12-17:12:57:58:INFO:sockeye.training:__call__] E=33 B=133450	s/sec=2079.77 tok/sec=98472.52 u/sec=23.55	ppl=6.954082 
[2023-12-17:12:58:00:INFO:sockeye.training:__call__] E=33 B=133500	s/sec=2241.99 tok/sec=98301.10 u/sec=23.55	ppl=6.962445 
[2023-12-17:12:58:02:INFO:sockeye.training:__call__] E=33 B=133550	s/sec=2177.17 tok/sec=97126.13 u/sec=23.62	ppl=6.967945 
[2023-12-17:12:58:04:INFO:sockeye.training:__call__] E=33 B=133600	s/sec=2060.15 tok/sec=99304.33 u/sec=23.28	ppl=6.972753 
[2023-12-17:12:58:07:INFO:sockeye.training:__call__] E=33 B=133650	s/sec=2463.03 tok/sec=96389.46 u/sec=23.79	ppl=6.987339 
[2023-12-17:12:58:09:INFO:sockeye.training:__call__] E=33 B=133700	s/sec=1815.07 tok/sec=104218.24 u/sec=22.60	ppl=6.990759 
[2023-12-17:12:58:11:INFO:sockeye.training:__call__] E=33 B=133750	s/sec=2112.83 tok/sec=99417.13 u/sec=22.97	ppl=7.000052 
[2023-12-17:12:58:13:INFO:sockeye.training:__call__] E=33 B=133800	s/sec=1891.42 tok/sec=103766.11 u/sec=22.43	ppl=7.004530 
[2023-12-17:12:58:15:INFO:sockeye.training:__call__] E=33 B=133850	s/sec=2091.77 tok/sec=99986.48 u/sec=23.35	ppl=7.012973 
[2023-12-17:12:58:18:INFO:sockeye.training:__call__] E=33 B=133900	s/sec=1973.64 tok/sec=102019.82 u/sec=22.68	ppl=7.018499 
[2023-12-17:12:58:20:INFO:sockeye.training:__call__] E=33 B=133950	s/sec=1932.99 tok/sec=100937.95 u/sec=22.75	ppl=7.021950 
[2023-12-17:12:58:22:INFO:sockeye.training:__call__] E=33 B=134000	s/sec=2235.17 tok/sec=97826.97 u/sec=23.24	ppl=7.030823 
[2023-12-17:12:58:24:INFO:sockeye.training:__call__] E=33 B=134050	s/sec=1913.44 tok/sec=99904.77 u/sec=23.50	ppl=7.035194 
[2023-12-17:12:58:26:INFO:sockeye.training:__call__] E=33 B=134100	s/sec=2142.30 tok/sec=95617.62 u/sec=24.08	ppl=7.039610 
[2023-12-17:12:58:28:INFO:sockeye.training:__call__] E=33 B=134150	s/sec=2039.17 tok/sec=98815.09 u/sec=23.56	ppl=7.045351 
[2023-12-17:12:58:30:INFO:sockeye.training:__call__] E=33 B=134200	s/sec=1997.89 tok/sec=98638.07 u/sec=23.52	ppl=7.050049 
[2023-12-17:12:58:33:INFO:sockeye.training:__call__] E=33 B=134250	s/sec=1898.05 tok/sec=102363.04 u/sec=22.90	ppl=7.055808 
[2023-12-17:12:58:35:INFO:sockeye.training:__call__] E=33 B=134300	s/sec=2278.32 tok/sec=96564.44 u/sec=23.58	ppl=7.064098 
[2023-12-17:12:58:37:INFO:sockeye.training:__call__] E=33 B=134350	s/sec=2000.95 tok/sec=99973.14 u/sec=23.25	ppl=7.070648 
[2023-12-17:12:58:39:INFO:sockeye.training:__call__] E=33 B=134400	s/sec=1725.29 tok/sec=105204.42 u/sec=21.78	ppl=7.075455 
[2023-12-17:12:58:41:INFO:sockeye.training:__call__] E=33 B=134450	s/sec=2106.33 tok/sec=97854.02 u/sec=23.51	ppl=7.083734 
[2023-12-17:12:58:43:INFO:sockeye.training:__call__] E=33 B=134500	s/sec=1898.64 tok/sec=100085.43 u/sec=23.54	ppl=7.087829 
[2023-12-17:12:58:46:INFO:sockeye.training:__call__] E=33 B=134550	s/sec=1855.18 tok/sec=99649.53 u/sec=23.33	ppl=7.092572 
[2023-12-17:12:58:48:INFO:sockeye.training:__call__] E=33 B=134600	s/sec=2062.26 tok/sec=101855.00 u/sec=22.85	ppl=7.099014 
[2023-12-17:12:58:50:INFO:sockeye.training:__call__] E=33 B=134650	s/sec=2009.05 tok/sec=98670.83 u/sec=23.30	ppl=7.104384 
[2023-12-17:12:58:52:INFO:sockeye.training:__call__] E=33 B=134700	s/sec=2034.17 tok/sec=100518.70 u/sec=22.95	ppl=7.111581 
[2023-12-17:12:58:54:INFO:sockeye.training:__call__] E=33 B=134750	s/sec=1921.43 tok/sec=100741.90 u/sec=23.09	ppl=7.118197 
[2023-12-17:12:58:56:INFO:sockeye.training:__call__] E=33 B=134800	s/sec=1866.64 tok/sec=103111.26 u/sec=22.65	ppl=7.121666 
[2023-12-17:12:58:59:INFO:sockeye.training:__call__] E=33 B=134850	s/sec=2005.88 tok/sec=101016.01 u/sec=23.26	ppl=7.130681 
[2023-12-17:12:59:01:INFO:sockeye.training:__call__] E=33 B=134900	s/sec=2129.93 tok/sec=98426.62 u/sec=23.44	ppl=7.138944 
[2023-12-17:12:59:03:INFO:sockeye.training:__call__] E=33 B=134950	s/sec=2229.61 tok/sec=95508.67 u/sec=23.98	ppl=7.146011 
[2023-12-17:12:59:05:INFO:sockeye.training:__call__] E=33 B=135000	s/sec=2451.47 tok/sec=93368.84 u/sec=23.79	ppl=7.155591 
[2023-12-17:12:59:07:INFO:sockeye.training:__call__] E=33 B=135050	s/sec=2033.47 tok/sec=98382.42 u/sec=23.32	ppl=7.161007 
[2023-12-17:12:59:09:INFO:sockeye.training:__call__] E=33 B=135100	s/sec=2108.99 tok/sec=101499.66 u/sec=23.29	ppl=7.166676 
[2023-12-17:12:59:11:INFO:sockeye.training:__call__] E=33 B=135150	s/sec=1918.51 tok/sec=100837.69 u/sec=23.01	ppl=7.169295 
[2023-12-17:12:59:13:INFO:sockeye.training:__call__] E=33 B=135200	s/sec=2111.18 tok/sec=100236.29 u/sec=23.23	ppl=7.174106 
[2023-12-17:12:59:16:INFO:sockeye.training:__call__] E=33 B=135250	s/sec=2118.64 tok/sec=99851.06 u/sec=23.23	ppl=7.180548 
[2023-12-17:12:59:18:INFO:sockeye.training:__call__] E=33 B=135300	s/sec=2128.70 tok/sec=98896.02 u/sec=23.10	ppl=7.185573 
[2023-12-17:12:59:20:INFO:sockeye.training:__call__] E=33 B=135350	s/sec=2217.60 tok/sec=97723.09 u/sec=23.26	ppl=7.191733 
[2023-12-17:12:59:22:INFO:sockeye.training:__call__] E=33 B=135400	s/sec=2152.13 tok/sec=100032.83 u/sec=23.43	ppl=7.196488 
[2023-12-17:12:59:24:INFO:sockeye.training:__call__] E=33 B=135450	s/sec=2273.89 tok/sec=96576.03 u/sec=23.61	ppl=7.204728 
[2023-12-17:12:59:26:INFO:sockeye.training:__call__] E=33 B=135500	s/sec=2137.60 tok/sec=97804.77 u/sec=23.44	ppl=7.210339 
[2023-12-17:12:59:28:INFO:sockeye.training:__call__] E=33 B=135550	s/sec=1968.16 tok/sec=98727.27 u/sec=23.75	ppl=7.214164 
[2023-12-17:12:59:31:INFO:sockeye.training:__call__] E=33 B=135600	s/sec=2293.05 tok/sec=97786.72 u/sec=23.46	ppl=7.220515 
[2023-12-17:12:59:33:INFO:sockeye.training:__call__] E=33 B=135650	s/sec=2154.30 tok/sec=96368.30 u/sec=23.50	ppl=7.227462 
[2023-12-17:12:59:35:INFO:sockeye.training:__call__] E=33 B=135700	s/sec=1770.91 tok/sec=102367.93 u/sec=22.92	ppl=7.231736 
[2023-12-17:12:59:37:INFO:sockeye.training:__call__] E=33 B=135750	s/sec=1951.52 tok/sec=101710.96 u/sec=22.67	ppl=7.236512 
[2023-12-17:12:59:39:INFO:sockeye.training:__call__] E=33 B=135800	s/sec=2096.07 tok/sec=97341.41 u/sec=23.44	ppl=7.242467 
[2023-12-17:12:59:42:INFO:sockeye.training:__call__] E=34 B=135850	s/sec=2001.38 tok/sec=96402.73 u/sec=21.91	ppl=7.240988 
[2023-12-17:12:59:44:INFO:sockeye.training:__call__] E=34 B=135900	s/sec=1920.30 tok/sec=103230.46 u/sec=22.73	ppl=7.229168 
[2023-12-17:12:59:46:INFO:sockeye.training:__call__] E=34 B=135950	s/sec=2132.43 tok/sec=98956.65 u/sec=23.30	ppl=7.218418 
[2023-12-17:12:59:48:INFO:sockeye.training:__call__] E=34 B=136000	s/sec=2077.02 tok/sec=100530.68 u/sec=23.26	ppl=7.208355 
[2023-12-17:12:59:48:INFO:sockeye.training:_create_checkpoint] Checkpoint [34]	Updates=136000 Epoch=34 Samples=12120552 Time-cost=172.245 Updates/sec=23.223
[2023-12-17:12:59:48:INFO:sockeye.training:_create_checkpoint] Checkpoint [34]	Train-ppl=7.208355
[2023-12-17:12:59:51:INFO:sockeye.training:_evaluate] Checkpoint [34]	Validation-ppl=30.158287	Validation-bleu=0.174623	Validation-chrf=32.169727	Validation-rouge1=0.175884	Validation-rouge2=0.084897	Validation-rougel=0.168130	Validation-length-ratio-mse=1.297719	Validation-ter=95.790343	Validation-avg-sec-per-sent=0.003459	Validation-decode-walltime=1.729492
[2023-12-17:12:59:51:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 10 checkpoints, best so far: 32.780032
[2023-12-17:12:59:51:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:12:59:51:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [34]	Learning-rate=0.000146
[2023-12-17:12:59:51:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:12:59:51:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:12:59:51:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00034"
[2023-12-17:12:59:54:INFO:sockeye.training:__call__] E=34 B=136050	s/sec=813.14 tok/sec=37892.84 u/sec=8.98	ppl=6.520297 
[2023-12-17:12:59:56:INFO:sockeye.training:__call__] E=34 B=136100	s/sec=2133.33 tok/sec=95354.76 u/sec=23.72	ppl=6.529392 
[2023-12-17:12:59:58:INFO:sockeye.training:__call__] E=34 B=136150	s/sec=1937.03 tok/sec=107240.58 u/sec=21.89	ppl=6.518262 
[2023-12-17:13:00:00:INFO:sockeye.training:__call__] E=34 B=136200	s/sec=2151.66 tok/sec=98773.81 u/sec=23.47	ppl=6.528977 
[2023-12-17:13:00:02:INFO:sockeye.training:__call__] E=34 B=136250	s/sec=1982.14 tok/sec=99456.54 u/sec=22.77	ppl=6.526840 
[2023-12-17:13:00:04:INFO:sockeye.training:__call__] E=34 B=136300	s/sec=1711.89 tok/sec=103720.92 u/sec=22.81	ppl=6.499696 
[2023-12-17:13:00:07:INFO:sockeye.training:__call__] E=34 B=136350	s/sec=2241.06 tok/sec=93142.12 u/sec=23.98	ppl=6.532026 
[2023-12-17:13:00:09:INFO:sockeye.training:__call__] E=34 B=136400	s/sec=2195.00 tok/sec=96967.11 u/sec=23.25	ppl=6.549340 
[2023-12-17:13:00:11:INFO:sockeye.training:__call__] E=34 B=136450	s/sec=2046.37 tok/sec=101076.54 u/sec=22.96	ppl=6.559263 
[2023-12-17:13:00:13:INFO:sockeye.training:__call__] E=34 B=136500	s/sec=2304.39 tok/sec=94629.54 u/sec=23.96	ppl=6.569294 
[2023-12-17:13:00:15:INFO:sockeye.training:__call__] E=34 B=136550	s/sec=1911.14 tok/sec=101646.34 u/sec=22.71	ppl=6.574305 
[2023-12-17:13:00:17:INFO:sockeye.training:__call__] E=34 B=136600	s/sec=2110.72 tok/sec=98735.00 u/sec=23.43	ppl=6.581764 
[2023-12-17:13:00:19:INFO:sockeye.training:__call__] E=34 B=136650	s/sec=2030.22 tok/sec=101109.32 u/sec=23.24	ppl=6.589138 
[2023-12-17:13:00:22:INFO:sockeye.training:__call__] E=34 B=136700	s/sec=1924.45 tok/sec=102247.73 u/sec=22.82	ppl=6.589911 
[2023-12-17:13:00:24:INFO:sockeye.training:__call__] E=34 B=136750	s/sec=2162.89 tok/sec=97782.18 u/sec=23.35	ppl=6.605959 
[2023-12-17:13:00:26:INFO:sockeye.training:__call__] E=34 B=136800	s/sec=2056.51 tok/sec=94468.98 u/sec=24.02	ppl=6.618749 
[2023-12-17:13:00:28:INFO:sockeye.training:__call__] E=34 B=136850	s/sec=2196.48 tok/sec=98325.80 u/sec=23.03	ppl=6.635925 
[2023-12-17:13:00:30:INFO:sockeye.training:__call__] E=34 B=136900	s/sec=2098.96 tok/sec=96951.57 u/sec=23.55	ppl=6.644747 
[2023-12-17:13:00:32:INFO:sockeye.training:__call__] E=34 B=136950	s/sec=2104.04 tok/sec=98222.87 u/sec=23.03	ppl=6.656243 
[2023-12-17:13:00:35:INFO:sockeye.training:__call__] E=34 B=137000	s/sec=1974.32 tok/sec=99747.51 u/sec=23.02	ppl=6.661831 
[2023-12-17:13:00:37:INFO:sockeye.training:__call__] E=34 B=137050	s/sec=1963.86 tok/sec=101532.29 u/sec=22.99	ppl=6.667812 
[2023-12-17:13:00:39:INFO:sockeye.training:__call__] E=34 B=137100	s/sec=2068.61 tok/sec=99469.59 u/sec=22.96	ppl=6.677125 
[2023-12-17:13:00:41:INFO:sockeye.training:__call__] E=34 B=137150	s/sec=2162.80 tok/sec=96340.29 u/sec=22.99	ppl=6.691556 
[2023-12-17:13:00:43:INFO:sockeye.training:__call__] E=34 B=137200	s/sec=2076.71 tok/sec=99474.25 u/sec=23.18	ppl=6.697477 
[2023-12-17:13:00:45:INFO:sockeye.training:__call__] E=34 B=137250	s/sec=1930.51 tok/sec=100416.32 u/sec=23.11	ppl=6.702583 
[2023-12-17:13:00:48:INFO:sockeye.training:__call__] E=34 B=137300	s/sec=2130.17 tok/sec=99851.61 u/sec=23.11	ppl=6.710946 
[2023-12-17:13:00:50:INFO:sockeye.training:__call__] E=34 B=137350	s/sec=1894.85 tok/sec=101543.38 u/sec=22.73	ppl=6.716456 
[2023-12-17:13:00:52:INFO:sockeye.training:__call__] E=34 B=137400	s/sec=2115.46 tok/sec=100873.36 u/sec=22.72	ppl=6.724599 
[2023-12-17:13:00:54:INFO:sockeye.training:__call__] E=34 B=137450	s/sec=1850.80 tok/sec=101057.96 u/sec=23.23	ppl=6.725272 
[2023-12-17:13:00:56:INFO:sockeye.training:__call__] E=34 B=137500	s/sec=1880.72 tok/sec=101477.96 u/sec=22.91	ppl=6.732438 
[2023-12-17:13:00:58:INFO:sockeye.training:__call__] E=34 B=137550	s/sec=1973.38 tok/sec=96201.45 u/sec=23.58	ppl=6.738172 
[2023-12-17:13:01:01:INFO:sockeye.training:__call__] E=34 B=137600	s/sec=1983.84 tok/sec=98879.79 u/sec=23.22	ppl=6.742607 
[2023-12-17:13:01:03:INFO:sockeye.training:__call__] E=34 B=137650	s/sec=2023.14 tok/sec=99163.62 u/sec=23.24	ppl=6.750508 
[2023-12-17:13:01:05:INFO:sockeye.training:__call__] E=34 B=137700	s/sec=1839.26 tok/sec=104043.25 u/sec=22.06	ppl=6.758937 
[2023-12-17:13:01:07:INFO:sockeye.training:__call__] E=34 B=137750	s/sec=2022.88 tok/sec=98991.69 u/sec=23.11	ppl=6.765262 
[2023-12-17:13:01:09:INFO:sockeye.training:__call__] E=34 B=137800	s/sec=2196.13 tok/sec=96419.32 u/sec=23.50	ppl=6.774126 
[2023-12-17:13:01:11:INFO:sockeye.training:__call__] E=34 B=137850	s/sec=2029.58 tok/sec=99739.29 u/sec=22.94	ppl=6.781237 
[2023-12-17:13:01:14:INFO:sockeye.training:__call__] E=34 B=137900	s/sec=2270.01 tok/sec=96813.67 u/sec=23.37	ppl=6.791395 
[2023-12-17:13:01:16:INFO:sockeye.training:__call__] E=34 B=137950	s/sec=1958.97 tok/sec=102113.80 u/sec=22.72	ppl=6.796189 
[2023-12-17:13:01:18:INFO:sockeye.training:__call__] E=34 B=138000	s/sec=2072.10 tok/sec=102372.52 u/sec=22.52	ppl=6.802502 
[2023-12-17:13:01:20:INFO:sockeye.training:__call__] E=34 B=138050	s/sec=2115.57 tok/sec=97756.28 u/sec=23.32	ppl=6.812168 
[2023-12-17:13:01:22:INFO:sockeye.training:__call__] E=34 B=138100	s/sec=2142.66 tok/sec=95924.84 u/sec=23.62	ppl=6.820795 
[2023-12-17:13:01:24:INFO:sockeye.training:__call__] E=34 B=138150	s/sec=2243.39 tok/sec=96488.44 u/sec=23.45	ppl=6.825976 
[2023-12-17:13:01:27:INFO:sockeye.training:__call__] E=34 B=138200	s/sec=2036.36 tok/sec=99774.16 u/sec=23.31	ppl=6.833162 
[2023-12-17:13:01:29:INFO:sockeye.training:__call__] E=34 B=138250	s/sec=1861.68 tok/sec=101397.86 u/sec=22.77	ppl=6.840238 
[2023-12-17:13:01:31:INFO:sockeye.training:__call__] E=34 B=138300	s/sec=2109.46 tok/sec=95899.48 u/sec=23.50	ppl=6.846689 
[2023-12-17:13:01:33:INFO:sockeye.training:__call__] E=34 B=138350	s/sec=2343.02 tok/sec=93659.34 u/sec=24.05	ppl=6.854399 
[2023-12-17:13:01:35:INFO:sockeye.training:__call__] E=34 B=138400	s/sec=1879.94 tok/sec=102407.90 u/sec=22.64	ppl=6.856661 
[2023-12-17:13:01:37:INFO:sockeye.training:__call__] E=34 B=138450	s/sec=2140.70 tok/sec=97547.82 u/sec=23.11	ppl=6.864717 
[2023-12-17:13:01:40:INFO:sockeye.training:__call__] E=34 B=138500	s/sec=1763.45 tok/sec=101972.53 u/sec=22.87	ppl=6.867440 
[2023-12-17:13:01:42:INFO:sockeye.training:__call__] E=34 B=138550	s/sec=2238.84 tok/sec=93877.05 u/sec=24.13	ppl=6.875731 
[2023-12-17:13:01:44:INFO:sockeye.training:__call__] E=34 B=138600	s/sec=1936.78 tok/sec=101920.07 u/sec=23.01	ppl=6.881959 
[2023-12-17:13:01:46:INFO:sockeye.training:__call__] E=34 B=138650	s/sec=2007.21 tok/sec=101364.24 u/sec=23.06	ppl=6.888349 
[2023-12-17:13:01:48:INFO:sockeye.training:__call__] E=34 B=138700	s/sec=2218.33 tok/sec=95139.94 u/sec=23.50	ppl=6.896233 
[2023-12-17:13:01:50:INFO:sockeye.training:__call__] E=34 B=138750	s/sec=2210.24 tok/sec=96479.36 u/sec=23.18	ppl=6.904839 
[2023-12-17:13:01:52:INFO:sockeye.training:__call__] E=34 B=138800	s/sec=2119.97 tok/sec=100391.53 u/sec=22.73	ppl=6.911518 
[2023-12-17:13:01:55:INFO:sockeye.training:__call__] E=34 B=138850	s/sec=2230.22 tok/sec=98249.45 u/sec=23.39	ppl=6.920710 
[2023-12-17:13:01:57:INFO:sockeye.training:__call__] E=34 B=138900	s/sec=1942.13 tok/sec=103967.75 u/sec=22.69	ppl=6.923752 
[2023-12-17:13:01:59:INFO:sockeye.training:__call__] E=34 B=138950	s/sec=2221.59 tok/sec=96428.43 u/sec=23.73	ppl=6.933174 
[2023-12-17:13:02:01:INFO:sockeye.training:__call__] E=34 B=139000	s/sec=1970.75 tok/sec=102110.81 u/sec=23.02	ppl=6.938540 
[2023-12-17:13:02:03:INFO:sockeye.training:__call__] E=34 B=139050	s/sec=2235.68 tok/sec=97429.42 u/sec=23.33	ppl=6.945179 
[2023-12-17:13:02:05:INFO:sockeye.training:__call__] E=34 B=139100	s/sec=2042.66 tok/sec=99508.79 u/sec=23.30	ppl=6.947931 
[2023-12-17:13:02:07:INFO:sockeye.training:__call__] E=34 B=139150	s/sec=2135.83 tok/sec=97808.40 u/sec=23.30	ppl=6.953195 
[2023-12-17:13:02:10:INFO:sockeye.training:__call__] E=34 B=139200	s/sec=1960.56 tok/sec=98806.12 u/sec=22.95	ppl=6.959543 
[2023-12-17:13:02:12:INFO:sockeye.training:__call__] E=34 B=139250	s/sec=2108.15 tok/sec=98690.48 u/sec=22.72	ppl=6.964947 
[2023-12-17:13:02:14:INFO:sockeye.training:__call__] E=34 B=139300	s/sec=2218.81 tok/sec=94780.68 u/sec=23.79	ppl=6.971198 
[2023-12-17:13:02:16:INFO:sockeye.training:__call__] E=34 B=139350	s/sec=1981.64 tok/sec=100334.01 u/sec=23.15	ppl=6.975912 
[2023-12-17:13:02:18:INFO:sockeye.training:__call__] E=34 B=139400	s/sec=2235.62 tok/sec=98128.72 u/sec=23.29	ppl=6.981174 
[2023-12-17:13:02:20:INFO:sockeye.training:__call__] E=34 B=139450	s/sec=2155.57 tok/sec=97389.81 u/sec=23.39	ppl=6.987570 
[2023-12-17:13:02:23:INFO:sockeye.training:__call__] E=34 B=139500	s/sec=2259.78 tok/sec=96843.41 u/sec=23.23	ppl=6.994605 
[2023-12-17:13:02:25:INFO:sockeye.training:__call__] E=34 B=139550	s/sec=1924.26 tok/sec=104141.50 u/sec=22.61	ppl=6.999259 
[2023-12-17:13:02:27:INFO:sockeye.training:__call__] E=34 B=139600	s/sec=2078.55 tok/sec=97088.61 u/sec=23.28	ppl=7.004989 
[2023-12-17:13:02:29:INFO:sockeye.training:__call__] E=34 B=139650	s/sec=2148.02 tok/sec=98496.62 u/sec=23.23	ppl=7.010396 
[2023-12-17:13:02:31:INFO:sockeye.training:__call__] E=34 B=139700	s/sec=2178.95 tok/sec=94966.68 u/sec=24.02	ppl=7.014914 
[2023-12-17:13:02:33:INFO:sockeye.training:__call__] E=34 B=139750	s/sec=2151.15 tok/sec=95092.33 u/sec=23.80	ppl=7.021917 
[2023-12-17:13:02:36:INFO:sockeye.training:__call__] E=34 B=139800	s/sec=1724.55 tok/sec=105331.05 u/sec=22.00	ppl=7.024543 
[2023-12-17:13:02:38:INFO:sockeye.training:__call__] E=35 B=139850	s/sec=1880.92 tok/sec=96971.69 u/sec=21.77	ppl=7.021213 
[2023-12-17:13:02:40:INFO:sockeye.training:__call__] E=35 B=139900	s/sec=1882.24 tok/sec=100464.40 u/sec=22.98	ppl=7.011102 
[2023-12-17:13:02:42:INFO:sockeye.training:__call__] E=35 B=139950	s/sec=2160.49 tok/sec=101091.70 u/sec=22.93	ppl=7.001555 
[2023-12-17:13:02:44:INFO:sockeye.training:__call__] E=35 B=140000	s/sec=2054.01 tok/sec=98800.99 u/sec=23.26	ppl=6.993155 
[2023-12-17:13:02:44:INFO:sockeye.training:_create_checkpoint] Checkpoint [35]	Updates=140000 Epoch=35 Samples=12476928 Time-cost=172.880 Updates/sec=23.137
[2023-12-17:13:02:44:INFO:sockeye.training:_create_checkpoint] Checkpoint [35]	Train-ppl=6.993155
[2023-12-17:13:02:47:INFO:sockeye.training:_evaluate] Checkpoint [35]	Validation-ppl=30.597186	Validation-bleu=0.174329	Validation-chrf=31.597160	Validation-rouge1=0.167264	Validation-rouge2=0.083191	Validation-rougel=0.160078	Validation-length-ratio-mse=1.287048	Validation-ter=96.698308	Validation-avg-sec-per-sent=0.003397	Validation-decode-walltime=1.698481
[2023-12-17:13:02:47:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 11 checkpoints, best so far: 32.780032
[2023-12-17:13:02:47:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:02:47:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [35]	Learning-rate=0.000146
[2023-12-17:13:02:47:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:02:47:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:02:48:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00035"
[2023-12-17:13:02:50:INFO:sockeye.training:__call__] E=35 B=140050	s/sec=778.09 tok/sec=41122.40 u/sec=8.99	ppl=6.258254 
[2023-12-17:13:02:52:INFO:sockeye.training:__call__] E=35 B=140100	s/sec=2121.38 tok/sec=96541.85 u/sec=23.59	ppl=6.278848 
[2023-12-17:13:02:54:INFO:sockeye.training:__call__] E=35 B=140150	s/sec=2095.84 tok/sec=100483.12 u/sec=22.90	ppl=6.322257 
[2023-12-17:13:02:56:INFO:sockeye.training:__call__] E=35 B=140200	s/sec=1858.01 tok/sec=100516.74 u/sec=23.00	ppl=6.323372 
[2023-12-17:13:02:58:INFO:sockeye.training:__call__] E=35 B=140250	s/sec=2073.29 tok/sec=98095.40 u/sec=23.65	ppl=6.350065 
[2023-12-17:13:03:01:INFO:sockeye.training:__call__] E=35 B=140300	s/sec=2081.76 tok/sec=96290.69 u/sec=23.66	ppl=6.367389 
[2023-12-17:13:03:03:INFO:sockeye.training:__call__] E=35 B=140350	s/sec=1984.95 tok/sec=99579.13 u/sec=23.02	ppl=6.380817 
[2023-12-17:13:03:05:INFO:sockeye.training:__call__] E=35 B=140400	s/sec=2146.28 tok/sec=98544.97 u/sec=23.01	ppl=6.399889 
[2023-12-17:13:03:07:INFO:sockeye.training:__call__] E=35 B=140450	s/sec=2356.86 tok/sec=95364.00 u/sec=23.64	ppl=6.414419 
[2023-12-17:13:03:09:INFO:sockeye.training:__call__] E=35 B=140500	s/sec=1941.25 tok/sec=101680.20 u/sec=22.98	ppl=6.411106 
[2023-12-17:13:03:11:INFO:sockeye.training:__call__] E=35 B=140550	s/sec=1928.53 tok/sec=101389.60 u/sec=22.74	ppl=6.412377 
[2023-12-17:13:03:14:INFO:sockeye.training:__call__] E=35 B=140600	s/sec=2114.72 tok/sec=99803.05 u/sec=22.95	ppl=6.421554 
[2023-12-17:13:03:16:INFO:sockeye.training:__call__] E=35 B=140650	s/sec=2045.95 tok/sec=99416.85 u/sec=23.08	ppl=6.430500 
[2023-12-17:13:03:18:INFO:sockeye.training:__call__] E=35 B=140700	s/sec=2180.76 tok/sec=97336.94 u/sec=23.50	ppl=6.437279 
[2023-12-17:13:03:20:INFO:sockeye.training:__call__] E=35 B=140750	s/sec=2286.89 tok/sec=97994.45 u/sec=23.47	ppl=6.456258 
[2023-12-17:13:03:22:INFO:sockeye.training:__call__] E=35 B=140800	s/sec=2069.18 tok/sec=100498.70 u/sec=23.01	ppl=6.466013 
[2023-12-17:13:03:24:INFO:sockeye.training:__call__] E=35 B=140850	s/sec=1965.91 tok/sec=99102.33 u/sec=22.71	ppl=6.475937 
[2023-12-17:13:03:27:INFO:sockeye.training:__call__] E=35 B=140900	s/sec=1832.01 tok/sec=103020.97 u/sec=22.32	ppl=6.479906 
[2023-12-17:13:03:29:INFO:sockeye.training:__call__] E=35 B=140950	s/sec=1872.49 tok/sec=102231.31 u/sec=22.81	ppl=6.487321 
[2023-12-17:13:03:31:INFO:sockeye.training:__call__] E=35 B=141000	s/sec=2045.24 tok/sec=98467.96 u/sec=23.16	ppl=6.493984 
[2023-12-17:13:03:33:INFO:sockeye.training:__call__] E=35 B=141050	s/sec=2031.48 tok/sec=100209.78 u/sec=22.92	ppl=6.503850 
[2023-12-17:13:03:35:INFO:sockeye.training:__call__] E=35 B=141100	s/sec=2126.30 tok/sec=98325.41 u/sec=23.36	ppl=6.511176 
[2023-12-17:13:03:37:INFO:sockeye.training:__call__] E=35 B=141150	s/sec=2136.51 tok/sec=96207.24 u/sec=23.72	ppl=6.525775 
[2023-12-17:13:03:40:INFO:sockeye.training:__call__] E=35 B=141200	s/sec=2154.51 tok/sec=98928.08 u/sec=23.38	ppl=6.534334 
[2023-12-17:13:03:42:INFO:sockeye.training:__call__] E=35 B=141250	s/sec=2363.87 tok/sec=93265.26 u/sec=23.98	ppl=6.550126 
[2023-12-17:13:03:44:INFO:sockeye.training:__call__] E=35 B=141300	s/sec=1970.40 tok/sec=99441.02 u/sec=23.59	ppl=6.551810 
[2023-12-17:13:03:46:INFO:sockeye.training:__call__] E=35 B=141350	s/sec=2291.45 tok/sec=94778.27 u/sec=23.44	ppl=6.562912 
[2023-12-17:13:03:48:INFO:sockeye.training:__call__] E=35 B=141400	s/sec=2116.28 tok/sec=97242.46 u/sec=23.70	ppl=6.571601 
[2023-12-17:13:03:50:INFO:sockeye.training:__call__] E=35 B=141450	s/sec=1940.07 tok/sec=101268.57 u/sec=23.14	ppl=6.579787 
[2023-12-17:13:03:52:INFO:sockeye.training:__call__] E=35 B=141500	s/sec=2030.33 tok/sec=99705.14 u/sec=22.82	ppl=6.586924 
[2023-12-17:13:03:54:INFO:sockeye.training:__call__] E=35 B=141550	s/sec=2279.43 tok/sec=96962.65 u/sec=23.67	ppl=6.597755 
[2023-12-17:13:03:57:INFO:sockeye.training:__call__] E=35 B=141600	s/sec=2015.53 tok/sec=101302.34 u/sec=22.82	ppl=6.602822 
[2023-12-17:13:03:59:INFO:sockeye.training:__call__] E=35 B=141650	s/sec=1793.45 tok/sec=103360.86 u/sec=22.64	ppl=6.606318 
[2023-12-17:13:04:01:INFO:sockeye.training:__call__] E=35 B=141700	s/sec=1986.10 tok/sec=99155.64 u/sec=23.33	ppl=6.611344 
[2023-12-17:13:04:03:INFO:sockeye.training:__call__] E=35 B=141750	s/sec=2184.72 tok/sec=93805.55 u/sec=23.87	ppl=6.620325 
[2023-12-17:13:04:05:INFO:sockeye.training:__call__] E=35 B=141800	s/sec=2126.09 tok/sec=100686.00 u/sec=22.71	ppl=6.624357 
[2023-12-17:13:04:07:INFO:sockeye.training:__call__] E=35 B=141850	s/sec=2231.77 tok/sec=96364.33 u/sec=23.48	ppl=6.631917 
[2023-12-17:13:04:10:INFO:sockeye.training:__call__] E=35 B=141900	s/sec=2065.71 tok/sec=98740.88 u/sec=23.05	ppl=6.637748 
[2023-12-17:13:04:12:INFO:sockeye.training:__call__] E=35 B=141950	s/sec=2001.61 tok/sec=97768.99 u/sec=23.60	ppl=6.642443 
[2023-12-17:13:04:14:INFO:sockeye.training:__call__] E=35 B=142000	s/sec=2107.12 tok/sec=95703.06 u/sec=23.48	ppl=6.650041 
[2023-12-17:13:04:16:INFO:sockeye.training:__call__] E=35 B=142050	s/sec=2243.95 tok/sec=96773.36 u/sec=23.03	ppl=6.659436 
[2023-12-17:13:04:18:INFO:sockeye.training:__call__] E=35 B=142100	s/sec=1936.88 tok/sec=99695.42 u/sec=23.15	ppl=6.668565 
[2023-12-17:13:04:20:INFO:sockeye.training:__call__] E=35 B=142150	s/sec=2353.05 tok/sec=95581.88 u/sec=23.76	ppl=6.677813 
[2023-12-17:13:04:22:INFO:sockeye.training:__call__] E=35 B=142200	s/sec=1954.79 tok/sec=102132.96 u/sec=22.92	ppl=6.680785 
[2023-12-17:13:04:25:INFO:sockeye.training:__call__] E=35 B=142250	s/sec=1766.43 tok/sec=103061.37 u/sec=21.86	ppl=6.681506 
[2023-12-17:13:04:27:INFO:sockeye.training:__call__] E=35 B=142300	s/sec=1920.40 tok/sec=101795.61 u/sec=22.91	ppl=6.686783 
[2023-12-17:13:04:29:INFO:sockeye.training:__call__] E=35 B=142350	s/sec=2156.96 tok/sec=96487.17 u/sec=23.53	ppl=6.694574 
[2023-12-17:13:04:31:INFO:sockeye.training:__call__] E=35 B=142400	s/sec=2450.29 tok/sec=90852.18 u/sec=24.42	ppl=6.703117 
[2023-12-17:13:04:33:INFO:sockeye.training:__call__] E=35 B=142450	s/sec=2028.51 tok/sec=99059.44 u/sec=23.18	ppl=6.706035 
[2023-12-17:13:04:35:INFO:sockeye.training:__call__] E=35 B=142500	s/sec=1774.34 tok/sec=103043.31 u/sec=22.82	ppl=6.709621 
[2023-12-17:13:04:38:INFO:sockeye.training:__call__] E=35 B=142550	s/sec=2019.05 tok/sec=100369.24 u/sec=23.07	ppl=6.713820 
[2023-12-17:13:04:40:INFO:sockeye.training:__call__] E=35 B=142600	s/sec=2261.59 tok/sec=95934.16 u/sec=23.88	ppl=6.722011 
[2023-12-17:13:04:42:INFO:sockeye.training:__call__] E=35 B=142650	s/sec=1805.69 tok/sec=104476.90 u/sec=22.62	ppl=6.725031 
[2023-12-17:13:04:44:INFO:sockeye.training:__call__] E=35 B=142700	s/sec=2065.42 tok/sec=99523.84 u/sec=23.05	ppl=6.730202 
[2023-12-17:13:04:46:INFO:sockeye.training:__call__] E=35 B=142750	s/sec=2170.58 tok/sec=98826.70 u/sec=23.27	ppl=6.735371 
[2023-12-17:13:04:48:INFO:sockeye.training:__call__] E=35 B=142800	s/sec=2085.88 tok/sec=98632.47 u/sec=23.28	ppl=6.741952 
[2023-12-17:13:04:51:INFO:sockeye.training:__call__] E=35 B=142850	s/sec=1880.36 tok/sec=103345.52 u/sec=23.13	ppl=6.746024 
[2023-12-17:13:04:53:INFO:sockeye.training:__call__] E=35 B=142900	s/sec=2032.28 tok/sec=101326.32 u/sec=23.05	ppl=6.750380 
[2023-12-17:13:04:55:INFO:sockeye.training:__call__] E=35 B=142950	s/sec=2127.67 tok/sec=99490.90 u/sec=23.41	ppl=6.754632 
[2023-12-17:13:04:57:INFO:sockeye.training:__call__] E=35 B=143000	s/sec=1992.22 tok/sec=100281.18 u/sec=23.02	ppl=6.760008 
[2023-12-17:13:04:59:INFO:sockeye.training:__call__] E=35 B=143050	s/sec=2008.82 tok/sec=100677.28 u/sec=23.08	ppl=6.764904 
[2023-12-17:13:05:01:INFO:sockeye.training:__call__] E=35 B=143100	s/sec=2189.66 tok/sec=94831.86 u/sec=23.88	ppl=6.771350 
[2023-12-17:13:05:03:INFO:sockeye.training:__call__] E=35 B=143150	s/sec=2053.39 tok/sec=97827.87 u/sec=22.96	ppl=6.776887 
[2023-12-17:13:05:06:INFO:sockeye.training:__call__] E=35 B=143200	s/sec=2294.12 tok/sec=97279.60 u/sec=23.54	ppl=6.784014 
[2023-12-17:13:05:08:INFO:sockeye.training:__call__] E=35 B=143250	s/sec=2121.97 tok/sec=98171.88 u/sec=23.39	ppl=6.789062 
[2023-12-17:13:05:10:INFO:sockeye.training:__call__] E=35 B=143300	s/sec=2072.13 tok/sec=99051.36 u/sec=22.92	ppl=6.794418 
[2023-12-17:13:05:12:INFO:sockeye.training:__call__] E=35 B=143350	s/sec=2270.56 tok/sec=95629.79 u/sec=23.77	ppl=6.801065 
[2023-12-17:13:05:14:INFO:sockeye.training:__call__] E=35 B=143400	s/sec=2102.78 tok/sec=98487.17 u/sec=23.59	ppl=6.806920 
[2023-12-17:13:05:16:INFO:sockeye.training:__call__] E=35 B=143450	s/sec=1855.48 tok/sec=105691.70 u/sec=22.13	ppl=6.810731 
[2023-12-17:13:05:19:INFO:sockeye.training:__call__] E=35 B=143500	s/sec=2064.79 tok/sec=98809.46 u/sec=23.46	ppl=6.815717 
[2023-12-17:13:05:21:INFO:sockeye.training:__call__] E=35 B=143550	s/sec=2277.99 tok/sec=95135.64 u/sec=23.77	ppl=6.820380 
[2023-12-17:13:05:23:INFO:sockeye.training:__call__] E=35 B=143600	s/sec=1752.21 tok/sec=108644.31 u/sec=21.86	ppl=6.822991 
[2023-12-17:13:05:25:INFO:sockeye.training:__call__] E=35 B=143650	s/sec=2061.00 tok/sec=99196.01 u/sec=23.25	ppl=6.827204 
[2023-12-17:13:05:27:INFO:sockeye.training:__call__] E=35 B=143700	s/sec=2043.14 tok/sec=98219.35 u/sec=23.22	ppl=6.831806 
[2023-12-17:13:05:29:INFO:sockeye.training:__call__] E=35 B=143750	s/sec=2142.04 tok/sec=99694.09 u/sec=23.24	ppl=6.836304 
[2023-12-17:13:05:32:INFO:sockeye.training:__call__] E=35 B=143800	s/sec=2088.08 tok/sec=99488.66 u/sec=23.10	ppl=6.842585 
[2023-12-17:13:05:34:INFO:sockeye.training:__call__] E=36 B=143850	s/sec=1968.36 tok/sec=92483.92 u/sec=22.61	ppl=6.839117 
[2023-12-17:13:05:36:INFO:sockeye.training:__call__] E=36 B=143900	s/sec=2037.04 tok/sec=98854.66 u/sec=22.73	ppl=6.829451 
[2023-12-17:13:05:38:INFO:sockeye.training:__call__] E=36 B=143950	s/sec=2120.85 tok/sec=98064.09 u/sec=23.54	ppl=6.820835 
[2023-12-17:13:05:40:INFO:sockeye.training:__call__] E=36 B=144000	s/sec=2145.60 tok/sec=98431.28 u/sec=23.12	ppl=6.812174 
[2023-12-17:13:05:40:INFO:sockeye.training:_create_checkpoint] Checkpoint [36]	Updates=144000 Epoch=36 Samples=12833768 Time-cost=172.605 Updates/sec=23.174
[2023-12-17:13:05:40:INFO:sockeye.training:_create_checkpoint] Checkpoint [36]	Train-ppl=6.812174
[2023-12-17:13:05:43:INFO:sockeye.training:_evaluate] Checkpoint [36]	Validation-ppl=31.945002	Validation-bleu=0.165058	Validation-chrf=31.291654	Validation-rouge1=0.161237	Validation-rouge2=0.074632	Validation-rougel=0.154145	Validation-length-ratio-mse=1.273020	Validation-ter=96.120512	Validation-avg-sec-per-sent=0.003518	Validation-decode-walltime=1.758882
[2023-12-17:13:05:43:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 12 checkpoints, best so far: 32.780032
[2023-12-17:13:05:43:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:05:43:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [36]	Learning-rate=0.000146
[2023-12-17:13:05:43:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:05:43:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:05:44:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00036"
[2023-12-17:13:05:46:INFO:sockeye.training:__call__] E=36 B=144050	s/sec=758.54 tok/sec=39171.26 u/sec=8.88	ppl=6.156157 
[2023-12-17:13:05:48:INFO:sockeye.training:__call__] E=36 B=144100	s/sec=2037.86 tok/sec=98559.64 u/sec=23.20	ppl=6.185276 
[2023-12-17:13:05:50:INFO:sockeye.training:__call__] E=36 B=144150	s/sec=1867.96 tok/sec=101870.40 u/sec=22.49	ppl=6.167588 
[2023-12-17:13:05:52:INFO:sockeye.training:__call__] E=36 B=144200	s/sec=2158.63 tok/sec=99400.87 u/sec=23.18	ppl=6.181231 
[2023-12-17:13:05:55:INFO:sockeye.training:__call__] E=36 B=144250	s/sec=1832.99 tok/sec=100535.58 u/sec=22.91	ppl=6.173321 
[2023-12-17:13:05:57:INFO:sockeye.training:__call__] E=36 B=144300	s/sec=2304.68 tok/sec=95214.37 u/sec=23.89	ppl=6.223812 
[2023-12-17:13:05:59:INFO:sockeye.training:__call__] E=36 B=144350	s/sec=2224.95 tok/sec=99034.11 u/sec=23.06	ppl=6.243394 
[2023-12-17:13:06:01:INFO:sockeye.training:__call__] E=36 B=144400	s/sec=1954.71 tok/sec=101820.64 u/sec=22.88	ppl=6.238646 
[2023-12-17:13:06:03:INFO:sockeye.training:__call__] E=36 B=144450	s/sec=2167.75 tok/sec=99612.78 u/sec=23.12	ppl=6.238784 
[2023-12-17:13:06:05:INFO:sockeye.training:__call__] E=36 B=144500	s/sec=2186.80 tok/sec=98070.35 u/sec=23.32	ppl=6.251659 
[2023-12-17:13:06:08:INFO:sockeye.training:__call__] E=36 B=144550	s/sec=2006.59 tok/sec=100322.00 u/sec=22.84	ppl=6.246144 
[2023-12-17:13:06:10:INFO:sockeye.training:__call__] E=36 B=144600	s/sec=2035.07 tok/sec=98907.92 u/sec=23.34	ppl=6.258827 
[2023-12-17:13:06:12:INFO:sockeye.training:__call__] E=36 B=144650	s/sec=1957.33 tok/sec=100654.10 u/sec=23.17	ppl=6.264692 
[2023-12-17:13:06:14:INFO:sockeye.training:__call__] E=36 B=144700	s/sec=2017.13 tok/sec=98421.24 u/sec=23.13	ppl=6.265950 
[2023-12-17:13:06:16:INFO:sockeye.training:__call__] E=36 B=144750	s/sec=1973.68 tok/sec=102039.86 u/sec=22.55	ppl=6.276620 
[2023-12-17:13:06:18:INFO:sockeye.training:__call__] E=36 B=144800	s/sec=2017.64 tok/sec=99144.00 u/sec=23.01	ppl=6.279135 
[2023-12-17:13:06:21:INFO:sockeye.training:__call__] E=36 B=144850	s/sec=2233.10 tok/sec=94842.12 u/sec=23.82	ppl=6.297447 
[2023-12-17:13:06:23:INFO:sockeye.training:__call__] E=36 B=144900	s/sec=2209.21 tok/sec=94259.49 u/sec=23.72	ppl=6.310557 
[2023-12-17:13:06:25:INFO:sockeye.training:__call__] E=36 B=144950	s/sec=2144.09 tok/sec=99245.10 u/sec=22.95	ppl=6.321881 
[2023-12-17:13:06:27:INFO:sockeye.training:__call__] E=36 B=145000	s/sec=2128.20 tok/sec=97717.22 u/sec=23.42	ppl=6.331375 
[2023-12-17:13:06:29:INFO:sockeye.training:__call__] E=36 B=145050	s/sec=2275.49 tok/sec=96597.88 u/sec=23.43	ppl=6.345865 
[2023-12-17:13:06:31:INFO:sockeye.training:__call__] E=36 B=145100	s/sec=2009.78 tok/sec=98689.05 u/sec=23.43	ppl=6.350082 
[2023-12-17:13:06:33:INFO:sockeye.training:__call__] E=36 B=145150	s/sec=2051.09 tok/sec=98242.88 u/sec=23.35	ppl=6.357061 
[2023-12-17:13:06:36:INFO:sockeye.training:__call__] E=36 B=145200	s/sec=1889.55 tok/sec=102548.66 u/sec=22.58	ppl=6.363937 
[2023-12-17:13:06:38:INFO:sockeye.training:__call__] E=36 B=145250	s/sec=2106.36 tok/sec=97967.92 u/sec=23.34	ppl=6.378145 
[2023-12-17:13:06:40:INFO:sockeye.training:__call__] E=36 B=145300	s/sec=2157.95 tok/sec=97293.21 u/sec=23.17	ppl=6.388750 
[2023-12-17:13:06:42:INFO:sockeye.training:__call__] E=36 B=145350	s/sec=2214.13 tok/sec=97675.47 u/sec=23.34	ppl=6.396938 
[2023-12-17:13:06:44:INFO:sockeye.training:__call__] E=36 B=145400	s/sec=1868.33 tok/sec=101775.35 u/sec=22.50	ppl=6.403127 
[2023-12-17:13:06:46:INFO:sockeye.training:__call__] E=36 B=145450	s/sec=2118.82 tok/sec=99230.86 u/sec=23.27	ppl=6.410203 
[2023-12-17:13:06:49:INFO:sockeye.training:__call__] E=36 B=145500	s/sec=2097.85 tok/sec=102386.62 u/sec=22.37	ppl=6.416993 
[2023-12-17:13:06:51:INFO:sockeye.training:__call__] E=36 B=145550	s/sec=2022.55 tok/sec=99045.66 u/sec=23.24	ppl=6.423152 
[2023-12-17:13:06:53:INFO:sockeye.training:__call__] E=36 B=145600	s/sec=1960.95 tok/sec=103475.32 u/sec=22.41	ppl=6.431179 
[2023-12-17:13:06:55:INFO:sockeye.training:__call__] E=36 B=145650	s/sec=1983.25 tok/sec=99695.70 u/sec=22.83	ppl=6.439560 
[2023-12-17:13:06:57:INFO:sockeye.training:__call__] E=36 B=145700	s/sec=2201.60 tok/sec=96930.71 u/sec=23.48	ppl=6.450180 
[2023-12-17:13:06:59:INFO:sockeye.training:__call__] E=36 B=145750	s/sec=1953.83 tok/sec=99645.20 u/sec=23.13	ppl=6.455053 
[2023-12-17:13:07:02:INFO:sockeye.training:__call__] E=36 B=145800	s/sec=1944.14 tok/sec=102731.45 u/sec=22.63	ppl=6.460812 
[2023-12-17:13:07:04:INFO:sockeye.training:__call__] E=36 B=145850	s/sec=2118.35 tok/sec=96804.29 u/sec=23.52	ppl=6.468146 
[2023-12-17:13:07:06:INFO:sockeye.training:__call__] E=36 B=145900	s/sec=2001.99 tok/sec=100445.01 u/sec=22.96	ppl=6.474177 
[2023-12-17:13:07:08:INFO:sockeye.training:__call__] E=36 B=145950	s/sec=1819.50 tok/sec=100756.80 u/sec=22.74	ppl=6.477368 
[2023-12-17:13:07:10:INFO:sockeye.training:__call__] E=36 B=146000	s/sec=2261.07 tok/sec=93859.31 u/sec=23.83	ppl=6.487441 
[2023-12-17:13:07:12:INFO:sockeye.training:__call__] E=36 B=146050	s/sec=1799.28 tok/sec=102198.93 u/sec=22.95	ppl=6.490054 
[2023-12-17:13:07:15:INFO:sockeye.training:__call__] E=36 B=146100	s/sec=2097.75 tok/sec=100514.88 u/sec=23.08	ppl=6.497113 
[2023-12-17:13:07:17:INFO:sockeye.training:__call__] E=36 B=146150	s/sec=2004.72 tok/sec=100102.83 u/sec=23.12	ppl=6.504079 
[2023-12-17:13:07:19:INFO:sockeye.training:__call__] E=36 B=146200	s/sec=1985.22 tok/sec=99968.84 u/sec=23.28	ppl=6.511512 
[2023-12-17:13:07:21:INFO:sockeye.training:__call__] E=36 B=146250	s/sec=2196.67 tok/sec=96638.30 u/sec=23.55	ppl=6.519710 
[2023-12-17:13:07:23:INFO:sockeye.training:__call__] E=36 B=146300	s/sec=2272.05 tok/sec=95937.58 u/sec=23.51	ppl=6.528350 
[2023-12-17:13:07:25:INFO:sockeye.training:__call__] E=36 B=146350	s/sec=2058.92 tok/sec=100588.91 u/sec=23.31	ppl=6.532503 
[2023-12-17:13:07:28:INFO:sockeye.training:__call__] E=36 B=146400	s/sec=2133.40 tok/sec=99519.26 u/sec=22.99	ppl=6.541593 
[2023-12-17:13:07:30:INFO:sockeye.training:__call__] E=36 B=146450	s/sec=2157.01 tok/sec=98238.16 u/sec=23.12	ppl=6.548271 
[2023-12-17:13:07:32:INFO:sockeye.training:__call__] E=36 B=146500	s/sec=2029.90 tok/sec=99972.69 u/sec=23.32	ppl=6.553356 
[2023-12-17:13:07:34:INFO:sockeye.training:__call__] E=36 B=146550	s/sec=2189.45 tok/sec=99224.93 u/sec=23.51	ppl=6.558838 
[2023-12-17:13:07:36:INFO:sockeye.training:__call__] E=36 B=146600	s/sec=2162.74 tok/sec=99544.93 u/sec=22.99	ppl=6.565025 
[2023-12-17:13:07:38:INFO:sockeye.training:__call__] E=36 B=146650	s/sec=2422.76 tok/sec=94332.32 u/sec=23.70	ppl=6.574968 
[2023-12-17:13:07:40:INFO:sockeye.training:__call__] E=36 B=146700	s/sec=2070.76 tok/sec=97757.83 u/sec=23.28	ppl=6.581144 
[2023-12-17:13:07:43:INFO:sockeye.training:__call__] E=36 B=146750	s/sec=2144.16 tok/sec=97527.75 u/sec=23.15	ppl=6.586556 
[2023-12-17:13:07:45:INFO:sockeye.training:__call__] E=36 B=146800	s/sec=1901.89 tok/sec=101189.32 u/sec=22.95	ppl=6.589838 
[2023-12-17:13:07:47:INFO:sockeye.training:__call__] E=36 B=146850	s/sec=2101.53 tok/sec=99869.06 u/sec=23.08	ppl=6.595216 
[2023-12-17:13:07:49:INFO:sockeye.training:__call__] E=36 B=146900	s/sec=2110.93 tok/sec=97808.95 u/sec=23.48	ppl=6.600367 
[2023-12-17:13:07:51:INFO:sockeye.training:__call__] E=36 B=146950	s/sec=2106.73 tok/sec=97863.98 u/sec=22.94	ppl=6.605348 
[2023-12-17:13:07:53:INFO:sockeye.training:__call__] E=36 B=147000	s/sec=2045.77 tok/sec=96466.34 u/sec=23.72	ppl=6.610571 
[2023-12-17:13:07:55:INFO:sockeye.training:__call__] E=36 B=147050	s/sec=1972.46 tok/sec=100510.20 u/sec=23.13	ppl=6.616345 
[2023-12-17:13:07:58:INFO:sockeye.training:__call__] E=36 B=147100	s/sec=1824.99 tok/sec=101012.66 u/sec=23.18	ppl=6.619268 
[2023-12-17:13:08:00:INFO:sockeye.training:__call__] E=36 B=147150	s/sec=2137.78 tok/sec=97756.70 u/sec=23.28	ppl=6.626717 
[2023-12-17:13:08:02:INFO:sockeye.training:__call__] E=36 B=147200	s/sec=1943.90 tok/sec=102302.89 u/sec=22.97	ppl=6.630296 
[2023-12-17:13:08:04:INFO:sockeye.training:__call__] E=36 B=147250	s/sec=1973.19 tok/sec=102679.56 u/sec=23.05	ppl=6.634916 
[2023-12-17:13:08:06:INFO:sockeye.training:__call__] E=36 B=147300	s/sec=2148.66 tok/sec=98061.05 u/sec=22.92	ppl=6.640551 
[2023-12-17:13:08:08:INFO:sockeye.training:__call__] E=36 B=147350	s/sec=2060.29 tok/sec=99455.22 u/sec=23.08	ppl=6.644987 
[2023-12-17:13:08:11:INFO:sockeye.training:__call__] E=36 B=147400	s/sec=1912.91 tok/sec=102463.19 u/sec=22.47	ppl=6.647919 
[2023-12-17:13:08:13:INFO:sockeye.training:__call__] E=36 B=147450	s/sec=1768.24 tok/sec=104372.67 u/sec=22.24	ppl=6.652110 
[2023-12-17:13:08:15:INFO:sockeye.training:__call__] E=36 B=147500	s/sec=2022.72 tok/sec=101403.84 u/sec=23.24	ppl=6.657902 
[2023-12-17:13:08:17:INFO:sockeye.training:__call__] E=36 B=147550	s/sec=2071.09 tok/sec=97144.56 u/sec=23.62	ppl=6.662457 
[2023-12-17:13:08:19:INFO:sockeye.training:__call__] E=36 B=147600	s/sec=2169.13 tok/sec=96241.06 u/sec=23.13	ppl=6.668286 
[2023-12-17:13:08:21:INFO:sockeye.training:__call__] E=36 B=147650	s/sec=2243.92 tok/sec=95086.98 u/sec=23.93	ppl=6.672907 
[2023-12-17:13:08:24:INFO:sockeye.training:__call__] E=36 B=147700	s/sec=2160.59 tok/sec=95370.48 u/sec=23.77	ppl=6.679366 
[2023-12-17:13:08:26:INFO:sockeye.training:__call__] E=36 B=147750	s/sec=2083.43 tok/sec=97391.88 u/sec=23.46	ppl=6.683600 
[2023-12-17:13:08:28:INFO:sockeye.training:__call__] E=36 B=147800	s/sec=1803.15 tok/sec=101772.84 u/sec=23.05	ppl=6.686893 
[2023-12-17:13:08:30:INFO:sockeye.training:__call__] E=37 B=147850	s/sec=2077.60 tok/sec=95313.65 u/sec=22.23	ppl=6.682727 
[2023-12-17:13:08:32:INFO:sockeye.training:__call__] E=37 B=147900	s/sec=2194.54 tok/sec=98754.35 u/sec=23.49	ppl=6.675163 
[2023-12-17:13:08:34:INFO:sockeye.training:__call__] E=37 B=147950	s/sec=2427.90 tok/sec=89777.54 u/sec=24.40	ppl=6.669977 
[2023-12-17:13:08:36:INFO:sockeye.training:__call__] E=37 B=148000	s/sec=1893.48 tok/sec=101648.51 u/sec=22.85	ppl=6.662224 
[2023-12-17:13:08:36:INFO:sockeye.training:_create_checkpoint] Checkpoint [37]	Updates=148000 Epoch=37 Samples=13190544 Time-cost=172.766 Updates/sec=23.153
[2023-12-17:13:08:36:INFO:sockeye.training:_create_checkpoint] Checkpoint [37]	Train-ppl=6.662224
[2023-12-17:13:08:39:INFO:sockeye.training:_evaluate] Checkpoint [37]	Validation-ppl=31.825074	Validation-bleu=0.160289	Validation-chrf=30.138325	Validation-rouge1=0.153954	Validation-rouge2=0.067648	Validation-rougel=0.147759	Validation-length-ratio-mse=1.267349	Validation-ter=96.574494	Validation-avg-sec-per-sent=0.003218	Validation-decode-walltime=1.609221
[2023-12-17:13:08:39:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 13 checkpoints, best so far: 32.780032
[2023-12-17:13:08:39:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:08:39:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [37]	Learning-rate=0.000146
[2023-12-17:13:08:40:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:08:40:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=5, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:08:40:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00037"
[2023-12-17:13:08:42:INFO:sockeye.training:__call__] E=37 B=148050	s/sec=755.60 tok/sec=40944.67 u/sec=9.06	ppl=5.988557 
[2023-12-17:13:08:44:INFO:sockeye.training:__call__] E=37 B=148100	s/sec=2127.81 tok/sec=99927.18 u/sec=22.69	ppl=6.051318 
[2023-12-17:13:08:46:INFO:sockeye.training:__call__] E=37 B=148150	s/sec=2008.49 tok/sec=100622.19 u/sec=22.87	ppl=6.047136 
[2023-12-17:13:08:49:INFO:sockeye.training:__call__] E=37 B=148200	s/sec=1900.55 tok/sec=101499.81 u/sec=22.93	ppl=6.043289 
[2023-12-17:13:08:51:INFO:sockeye.training:__call__] E=37 B=148250	s/sec=1915.67 tok/sec=98237.95 u/sec=23.38	ppl=6.049184 
[2023-12-17:13:08:53:INFO:sockeye.training:__call__] E=37 B=148300	s/sec=2424.39 tok/sec=95690.59 u/sec=23.90	ppl=6.071187 
[2023-12-17:13:08:55:INFO:sockeye.training:__call__] E=37 B=148350	s/sec=2415.95 tok/sec=93143.39 u/sec=24.16	ppl=6.106497 
[2023-12-17:13:08:57:INFO:sockeye.training:__call__] E=37 B=148400	s/sec=1989.30 tok/sec=102439.95 u/sec=22.73	ppl=6.114837 
[2023-12-17:13:08:59:INFO:sockeye.training:__call__] E=37 B=148450	s/sec=2061.72 tok/sec=97617.92 u/sec=23.34	ppl=6.128035 
[2023-12-17:13:09:01:INFO:sockeye.training:__call__] E=37 B=148500	s/sec=2024.19 tok/sec=102568.70 u/sec=22.59	ppl=6.132176 
[2023-12-17:13:09:04:INFO:sockeye.training:__call__] E=37 B=148550	s/sec=1853.06 tok/sec=99635.83 u/sec=22.75	ppl=6.141472 
[2023-12-17:13:09:06:INFO:sockeye.training:__call__] E=37 B=148600	s/sec=2079.25 tok/sec=99024.65 u/sec=23.41	ppl=6.153055 
[2023-12-17:13:09:08:INFO:sockeye.training:__call__] E=37 B=148650	s/sec=2175.57 tok/sec=97665.70 u/sec=23.69	ppl=6.166319 
[2023-12-17:13:09:10:INFO:sockeye.training:__call__] E=37 B=148700	s/sec=2016.18 tok/sec=100145.67 u/sec=23.29	ppl=6.167729 
[2023-12-17:13:09:12:INFO:sockeye.training:__call__] E=37 B=148750	s/sec=2059.78 tok/sec=99047.51 u/sec=23.20	ppl=6.176261 
[2023-12-17:13:09:14:INFO:sockeye.training:__call__] E=37 B=148800	s/sec=2246.75 tok/sec=96844.91 u/sec=23.29	ppl=6.187746 
[2023-12-17:13:09:17:INFO:sockeye.training:__call__] E=37 B=148850	s/sec=2168.12 tok/sec=103957.33 u/sec=21.93	ppl=6.200288 
[2023-12-17:13:09:19:INFO:sockeye.training:__call__] E=37 B=148900	s/sec=2007.65 tok/sec=100264.40 u/sec=23.07	ppl=6.206257 
[2023-12-17:13:09:21:INFO:sockeye.training:__call__] E=37 B=148950	s/sec=2073.06 tok/sec=102169.56 u/sec=22.61	ppl=6.211338 
[2023-12-17:13:09:23:INFO:sockeye.training:__call__] E=37 B=149000	s/sec=2051.73 tok/sec=101076.31 u/sec=23.02	ppl=6.222663 
[2023-12-17:13:09:25:INFO:sockeye.training:__call__] E=37 B=149050	s/sec=1798.97 tok/sec=103934.14 u/sec=22.67	ppl=6.223063 
[2023-12-17:13:09:28:INFO:sockeye.training:__call__] E=37 B=149100	s/sec=1806.47 tok/sec=104490.59 u/sec=22.81	ppl=6.225546 
[2023-12-17:13:09:30:INFO:sockeye.training:__call__] E=37 B=149150	s/sec=1958.32 tok/sec=99251.27 u/sec=23.05	ppl=6.237134 
[2023-12-17:13:09:32:INFO:sockeye.training:__call__] E=37 B=149200	s/sec=2224.03 tok/sec=98412.27 u/sec=23.44	ppl=6.247454 
[2023-12-17:13:09:34:INFO:sockeye.training:__call__] E=37 B=149250	s/sec=2151.28 tok/sec=97231.98 u/sec=23.67	ppl=6.255501 
[2023-12-17:13:09:36:INFO:sockeye.training:__call__] E=37 B=149300	s/sec=1858.16 tok/sec=99679.45 u/sec=22.95	ppl=6.259985 
[2023-12-17:13:09:38:INFO:sockeye.training:__call__] E=37 B=149350	s/sec=2036.43 tok/sec=99338.79 u/sec=23.44	ppl=6.270770 
[2023-12-17:13:09:40:INFO:sockeye.training:__call__] E=37 B=149400	s/sec=2244.35 tok/sec=94392.79 u/sec=23.90	ppl=6.282257 
[2023-12-17:13:09:43:INFO:sockeye.training:__call__] E=37 B=149450	s/sec=1955.87 tok/sec=101923.72 u/sec=22.76	ppl=6.290507 
[2023-12-17:13:09:45:INFO:sockeye.training:__call__] E=37 B=149500	s/sec=2300.02 tok/sec=96147.54 u/sec=24.00	ppl=6.299549 
[2023-12-17:13:09:47:INFO:sockeye.training:__call__] E=37 B=149550	s/sec=1956.60 tok/sec=99291.69 u/sec=23.07	ppl=6.305806 
[2023-12-17:13:09:49:INFO:sockeye.training:__call__] E=37 B=149600	s/sec=2076.40 tok/sec=100143.58 u/sec=23.26	ppl=6.309709 
[2023-12-17:13:09:51:INFO:sockeye.training:__call__] E=37 B=149650	s/sec=2250.18 tok/sec=96027.74 u/sec=23.64	ppl=6.317112 
[2023-12-17:13:09:53:INFO:sockeye.training:__call__] E=37 B=149700	s/sec=2252.12 tok/sec=96589.27 u/sec=23.86	ppl=6.324897 
[2023-12-17:13:09:55:INFO:sockeye.training:__call__] E=37 B=149750	s/sec=2090.61 tok/sec=99605.32 u/sec=23.25	ppl=6.330030 
[2023-12-17:13:09:57:INFO:sockeye.training:__call__] E=37 B=149800	s/sec=1928.32 tok/sec=101272.32 u/sec=23.31	ppl=6.336519 
[2023-12-17:13:10:00:INFO:sockeye.training:__call__] E=37 B=149850	s/sec=1879.19 tok/sec=102571.10 u/sec=23.12	ppl=6.341645 
[2023-12-17:13:10:02:INFO:sockeye.training:__call__] E=37 B=149900	s/sec=2162.74 tok/sec=97054.22 u/sec=23.67	ppl=6.350870 
[2023-12-17:13:10:04:INFO:sockeye.training:__call__] E=37 B=149950	s/sec=1972.21 tok/sec=101652.28 u/sec=22.74	ppl=6.355555 
[2023-12-17:13:10:06:INFO:sockeye.training:__call__] E=37 B=150000	s/sec=2062.19 tok/sec=100811.77 u/sec=22.65	ppl=6.362810 
[2023-12-17:13:10:08:INFO:sockeye.training:__call__] E=37 B=150050	s/sec=2149.56 tok/sec=98413.67 u/sec=23.12	ppl=6.368929 
[2023-12-17:13:10:10:INFO:sockeye.training:__call__] E=37 B=150100	s/sec=2209.36 tok/sec=98004.48 u/sec=23.36	ppl=6.376596 
[2023-12-17:13:10:13:INFO:sockeye.training:__call__] E=37 B=150150	s/sec=2101.28 tok/sec=97002.73 u/sec=23.37	ppl=6.382475 
[2023-12-17:13:10:15:INFO:sockeye.training:__call__] E=37 B=150200	s/sec=1916.18 tok/sec=103575.81 u/sec=22.77	ppl=6.387451 
[2023-12-17:13:10:17:INFO:sockeye.training:__call__] E=37 B=150250	s/sec=1965.27 tok/sec=101562.17 u/sec=22.96	ppl=6.393141 
[2023-12-17:13:10:19:INFO:sockeye.training:__call__] E=37 B=150300	s/sec=1753.02 tok/sec=103744.08 u/sec=22.45	ppl=6.397177 
[2023-12-17:13:10:21:INFO:sockeye.training:__call__] E=37 B=150350	s/sec=1914.57 tok/sec=100966.19 u/sec=22.92	ppl=6.401187 
[2023-12-17:13:10:24:INFO:sockeye.training:__call__] E=37 B=150400	s/sec=2070.55 tok/sec=100892.02 u/sec=23.07	ppl=6.407309 
[2023-12-17:13:10:26:INFO:sockeye.training:__call__] E=37 B=150450	s/sec=2149.28 tok/sec=97117.76 u/sec=23.16	ppl=6.411630 
[2023-12-17:13:10:28:INFO:sockeye.training:__call__] E=37 B=150500	s/sec=1962.32 tok/sec=99365.27 u/sec=23.10	ppl=6.414666 
[2023-12-17:13:10:30:INFO:sockeye.training:__call__] E=37 B=150550	s/sec=2403.60 tok/sec=94718.62 u/sec=23.69	ppl=6.423537 
[2023-12-17:13:10:32:INFO:sockeye.training:__call__] E=37 B=150600	s/sec=1832.51 tok/sec=104624.92 u/sec=22.37	ppl=6.428201 
[2023-12-17:13:10:34:INFO:sockeye.training:__call__] E=37 B=150650	s/sec=2040.81 tok/sec=100836.38 u/sec=22.94	ppl=6.434028 
[2023-12-17:13:10:37:INFO:sockeye.training:__call__] E=37 B=150700	s/sec=1994.30 tok/sec=99951.24 u/sec=23.08	ppl=6.439023 
[2023-12-17:13:10:39:INFO:sockeye.training:__call__] E=37 B=150750	s/sec=2108.45 tok/sec=98141.48 u/sec=23.24	ppl=6.445550 
[2023-12-17:13:10:41:INFO:sockeye.training:__call__] E=37 B=150800	s/sec=1870.56 tok/sec=101515.68 u/sec=22.57	ppl=6.448960 
[2023-12-17:13:10:43:INFO:sockeye.training:__call__] E=37 B=150850	s/sec=2102.66 tok/sec=99304.80 u/sec=23.06	ppl=6.452112 
[2023-12-17:13:10:45:INFO:sockeye.training:__call__] E=37 B=150900	s/sec=2077.59 tok/sec=99369.27 u/sec=23.10	ppl=6.458119 
[2023-12-17:13:10:47:INFO:sockeye.training:__call__] E=37 B=150950	s/sec=2137.93 tok/sec=99199.97 u/sec=23.04	ppl=6.464053 
[2023-12-17:13:10:50:INFO:sockeye.training:__call__] E=37 B=151000	s/sec=1970.67 tok/sec=100475.44 u/sec=22.64	ppl=6.469190 
[2023-12-17:13:10:52:INFO:sockeye.training:__call__] E=37 B=151050	s/sec=2199.95 tok/sec=94548.22 u/sec=23.91	ppl=6.475233 
[2023-12-17:13:10:54:INFO:sockeye.training:__call__] E=37 B=151100	s/sec=1980.91 tok/sec=99007.81 u/sec=23.40	ppl=6.479659 
[2023-12-17:13:10:56:INFO:sockeye.training:__call__] E=37 B=151150	s/sec=1988.16 tok/sec=95734.59 u/sec=23.67	ppl=6.484230 
[2023-12-17:13:10:58:INFO:sockeye.training:__call__] E=37 B=151200	s/sec=2237.02 tok/sec=97327.36 u/sec=23.58	ppl=6.489573 
[2023-12-17:13:11:00:INFO:sockeye.training:__call__] E=37 B=151250	s/sec=2173.27 tok/sec=96542.68 u/sec=23.54	ppl=6.494921 
[2023-12-17:13:11:02:INFO:sockeye.training:__call__] E=37 B=151300	s/sec=1799.85 tok/sec=100165.73 u/sec=23.29	ppl=6.497680 
[2023-12-17:13:11:05:INFO:sockeye.training:__call__] E=37 B=151350	s/sec=1881.19 tok/sec=101231.72 u/sec=22.96	ppl=6.501218 
[2023-12-17:13:11:07:INFO:sockeye.training:__call__] E=37 B=151400	s/sec=2119.02 tok/sec=99030.13 u/sec=23.03	ppl=6.505892 
[2023-12-17:13:11:09:INFO:sockeye.training:__call__] E=37 B=151450	s/sec=2208.37 tok/sec=98357.33 u/sec=23.51	ppl=6.511343 
[2023-12-17:13:11:11:INFO:sockeye.training:__call__] E=37 B=151500	s/sec=2287.62 tok/sec=96139.91 u/sec=23.36	ppl=6.517208 
[2023-12-17:13:11:13:INFO:sockeye.training:__call__] E=37 B=151550	s/sec=1912.40 tok/sec=102080.92 u/sec=22.51	ppl=6.520546 
[2023-12-17:13:11:15:INFO:sockeye.training:__call__] E=37 B=151600	s/sec=2258.89 tok/sec=93610.68 u/sec=23.77	ppl=6.526154 
[2023-12-17:13:11:17:INFO:sockeye.training:__call__] E=37 B=151650	s/sec=1934.08 tok/sec=100321.49 u/sec=23.02	ppl=6.530887 
[2023-12-17:13:11:20:INFO:sockeye.training:__call__] E=37 B=151700	s/sec=2086.11 tok/sec=97779.41 u/sec=23.58	ppl=6.533884 
[2023-12-17:13:11:22:INFO:sockeye.training:__call__] E=37 B=151750	s/sec=2137.90 tok/sec=97419.08 u/sec=23.48	ppl=6.538314 
[2023-12-17:13:11:24:INFO:sockeye.training:__call__] E=37 B=151800	s/sec=2442.23 tok/sec=90036.00 u/sec=24.31	ppl=6.544926 
[2023-12-17:13:11:26:INFO:sockeye.training:__call__] E=38 B=151850	s/sec=2100.68 tok/sec=94297.07 u/sec=22.10	ppl=6.540105 
[2023-12-17:13:11:28:INFO:sockeye.training:__call__] E=38 B=151900	s/sec=2152.31 tok/sec=99101.81 u/sec=22.99	ppl=6.532515 
[2023-12-17:13:11:30:INFO:sockeye.training:__call__] E=38 B=151950	s/sec=2157.73 tok/sec=96358.85 u/sec=23.09	ppl=6.525818 
[2023-12-17:13:11:33:INFO:sockeye.training:__call__] E=38 B=152000	s/sec=1892.60 tok/sec=100870.21 u/sec=22.97	ppl=6.518390 
[2023-12-17:13:11:33:INFO:sockeye.training:_create_checkpoint] Checkpoint [38]	Updates=152000 Epoch=38 Samples=13546832 Time-cost=172.806 Updates/sec=23.147
[2023-12-17:13:11:33:INFO:sockeye.training:_create_checkpoint] Checkpoint [38]	Train-ppl=6.518390
[2023-12-17:13:11:35:INFO:sockeye.training:_evaluate] Checkpoint [38]	Validation-ppl=32.037354	Validation-bleu=0.148973	Validation-chrf=29.653748	Validation-rouge1=0.155914	Validation-rouge2=0.067473	Validation-rougel=0.148545	Validation-length-ratio-mse=1.349364	Validation-ter=100.288898	Validation-avg-sec-per-sent=0.003524	Validation-decode-walltime=1.761959
[2023-12-17:13:11:35:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 14 checkpoints, best so far: 32.780032
[2023-12-17:13:11:35:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:11:35:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [38]	Learning-rate=0.000146
[2023-12-17:13:11:36:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:11:36:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=6, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:11:36:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00038"
[2023-12-17:13:11:38:INFO:sockeye.training:__call__] E=38 B=152050	s/sec=793.84 tok/sec=39754.99 u/sec=8.96	ppl=5.990198 
[2023-12-17:13:11:40:INFO:sockeye.training:__call__] E=38 B=152100	s/sec=1966.36 tok/sec=99873.66 u/sec=22.93	ppl=5.991153 
[2023-12-17:13:11:42:INFO:sockeye.training:__call__] E=38 B=152150	s/sec=2238.26 tok/sec=95529.25 u/sec=23.43	ppl=6.015067 
[2023-12-17:13:11:45:INFO:sockeye.training:__call__] E=38 B=152200	s/sec=2126.78 tok/sec=96783.61 u/sec=23.40	ppl=6.029489 
[2023-12-17:13:11:47:INFO:sockeye.training:__call__] E=38 B=152250	s/sec=1989.64 tok/sec=100639.96 u/sec=23.20	ppl=6.046742 
[2023-12-17:13:11:49:INFO:sockeye.training:__call__] E=38 B=152300	s/sec=2039.83 tok/sec=100095.45 u/sec=23.05	ppl=6.051387 
[2023-12-17:13:11:51:INFO:sockeye.training:__call__] E=38 B=152350	s/sec=2205.03 tok/sec=97547.20 u/sec=23.48	ppl=6.062183 
[2023-12-17:13:11:53:INFO:sockeye.training:__call__] E=38 B=152400	s/sec=1806.38 tok/sec=103287.58 u/sec=22.49	ppl=6.056680 
[2023-12-17:13:11:55:INFO:sockeye.training:__call__] E=38 B=152450	s/sec=2075.21 tok/sec=98370.18 u/sec=23.62	ppl=6.054616 
[2023-12-17:13:11:58:INFO:sockeye.training:__call__] E=38 B=152500	s/sec=1813.86 tok/sec=102219.87 u/sec=22.86	ppl=6.052842 
[2023-12-17:13:12:00:INFO:sockeye.training:__call__] E=38 B=152550	s/sec=1914.92 tok/sec=100413.65 u/sec=22.97	ppl=6.059517 
[2023-12-17:13:12:02:INFO:sockeye.training:__call__] E=38 B=152600	s/sec=1920.07 tok/sec=100923.89 u/sec=23.12	ppl=6.062629 
[2023-12-17:13:12:04:INFO:sockeye.training:__call__] E=38 B=152650	s/sec=2206.76 tok/sec=96072.30 u/sec=23.90	ppl=6.078364 
[2023-12-17:13:12:06:INFO:sockeye.training:__call__] E=38 B=152700	s/sec=1889.85 tok/sec=99980.12 u/sec=23.21	ppl=6.079116 
[2023-12-17:13:12:08:INFO:sockeye.training:__call__] E=38 B=152750	s/sec=1834.25 tok/sec=101424.00 u/sec=22.97	ppl=6.078409 
[2023-12-17:13:12:10:INFO:sockeye.training:__call__] E=38 B=152800	s/sec=2159.73 tok/sec=97656.10 u/sec=23.60	ppl=6.086129 
[2023-12-17:13:12:13:INFO:sockeye.training:__call__] E=38 B=152850	s/sec=1970.31 tok/sec=102426.54 u/sec=22.97	ppl=6.091862 
[2023-12-17:13:12:15:INFO:sockeye.training:__call__] E=38 B=152900	s/sec=2158.22 tok/sec=98559.95 u/sec=23.14	ppl=6.097931 
[2023-12-17:13:12:17:INFO:sockeye.training:__call__] E=38 B=152950	s/sec=2166.07 tok/sec=99477.83 u/sec=22.95	ppl=6.108328 
[2023-12-17:13:12:19:INFO:sockeye.training:__call__] E=38 B=153000	s/sec=2167.30 tok/sec=100167.54 u/sec=23.04	ppl=6.116980 
[2023-12-17:13:12:21:INFO:sockeye.training:__call__] E=38 B=153050	s/sec=1937.21 tok/sec=102583.95 u/sec=22.93	ppl=6.117473 
[2023-12-17:13:12:24:INFO:sockeye.training:__call__] E=38 B=153100	s/sec=2091.26 tok/sec=100467.34 u/sec=22.65	ppl=6.126290 
[2023-12-17:13:12:26:INFO:sockeye.training:__call__] E=38 B=153150	s/sec=1864.59 tok/sec=104272.87 u/sec=22.50	ppl=6.131490 
[2023-12-17:13:12:28:INFO:sockeye.training:__call__] E=38 B=153200	s/sec=2250.66 tok/sec=95896.47 u/sec=23.76	ppl=6.140835 
[2023-12-17:13:12:30:INFO:sockeye.training:__call__] E=38 B=153250	s/sec=2100.81 tok/sec=98570.24 u/sec=23.32	ppl=6.149899 
[2023-12-17:13:12:32:INFO:sockeye.training:__call__] E=38 B=153300	s/sec=2173.04 tok/sec=96970.66 u/sec=23.30	ppl=6.159085 
[2023-12-17:13:12:34:INFO:sockeye.training:__call__] E=38 B=153350	s/sec=2112.00 tok/sec=97766.08 u/sec=23.70	ppl=6.165797 
[2023-12-17:13:12:36:INFO:sockeye.training:__call__] E=38 B=153400	s/sec=2048.43 tok/sec=98415.84 u/sec=23.80	ppl=6.173840 
[2023-12-17:13:12:39:INFO:sockeye.training:__call__] E=38 B=153450	s/sec=1972.36 tok/sec=99401.17 u/sec=23.53	ppl=6.178195 
[2023-12-17:13:12:41:INFO:sockeye.training:__call__] E=38 B=153500	s/sec=2065.47 tok/sec=99557.84 u/sec=23.18	ppl=6.185679 
[2023-12-17:13:12:43:INFO:sockeye.training:__call__] E=38 B=153550	s/sec=2132.22 tok/sec=99017.47 u/sec=23.02	ppl=6.194299 
[2023-12-17:13:12:45:INFO:sockeye.training:__call__] E=38 B=153600	s/sec=2158.31 tok/sec=99364.29 u/sec=23.30	ppl=6.200011 
[2023-12-17:13:12:47:INFO:sockeye.training:__call__] E=38 B=153650	s/sec=2088.73 tok/sec=98425.44 u/sec=23.11	ppl=6.207631 
[2023-12-17:13:12:49:INFO:sockeye.training:__call__] E=38 B=153700	s/sec=2146.24 tok/sec=96569.63 u/sec=23.01	ppl=6.215494 
[2023-12-17:13:12:51:INFO:sockeye.training:__call__] E=38 B=153750	s/sec=2364.23 tok/sec=96445.26 u/sec=23.64	ppl=6.224153 
[2023-12-17:13:12:54:INFO:sockeye.training:__call__] E=38 B=153800	s/sec=2204.19 tok/sec=97736.58 u/sec=23.51	ppl=6.231516 
[2023-12-17:13:12:56:INFO:sockeye.training:__call__] E=38 B=153850	s/sec=2176.31 tok/sec=95727.30 u/sec=23.70	ppl=6.238247 
[2023-12-17:13:12:58:INFO:sockeye.training:__call__] E=38 B=153900	s/sec=2097.52 tok/sec=97336.93 u/sec=23.12	ppl=6.244979 
[2023-12-17:13:13:00:INFO:sockeye.training:__call__] E=38 B=153950	s/sec=1736.20 tok/sec=102943.33 u/sec=22.33	ppl=6.247524 
[2023-12-17:13:13:02:INFO:sockeye.training:__call__] E=38 B=154000	s/sec=2267.11 tok/sec=95984.44 u/sec=23.93	ppl=6.255939 
[2023-12-17:13:13:04:INFO:sockeye.training:__call__] E=38 B=154050	s/sec=1851.39 tok/sec=99019.28 u/sec=23.33	ppl=6.261112 
[2023-12-17:13:13:06:INFO:sockeye.training:__call__] E=38 B=154100	s/sec=2267.23 tok/sec=96815.09 u/sec=23.46	ppl=6.270112 
[2023-12-17:13:13:09:INFO:sockeye.training:__call__] E=38 B=154150	s/sec=1921.50 tok/sec=100629.97 u/sec=23.18	ppl=6.274035 
[2023-12-17:13:13:11:INFO:sockeye.training:__call__] E=38 B=154200	s/sec=1939.87 tok/sec=100531.79 u/sec=23.18	ppl=6.277202 
[2023-12-17:13:13:13:INFO:sockeye.training:__call__] E=38 B=154250	s/sec=2196.15 tok/sec=96285.19 u/sec=23.46	ppl=6.286572 
[2023-12-17:13:13:15:INFO:sockeye.training:__call__] E=38 B=154300	s/sec=2153.97 tok/sec=100086.94 u/sec=23.25	ppl=6.292031 
[2023-12-17:13:13:17:INFO:sockeye.training:__call__] E=38 B=154350	s/sec=2124.00 tok/sec=99036.56 u/sec=23.01	ppl=6.297995 
[2023-12-17:13:13:19:INFO:sockeye.training:__call__] E=38 B=154400	s/sec=2107.24 tok/sec=99485.64 u/sec=23.19	ppl=6.305432 
[2023-12-17:13:13:22:INFO:sockeye.training:__call__] E=38 B=154450	s/sec=2048.16 tok/sec=98282.02 u/sec=23.06	ppl=6.310971 
[2023-12-17:13:13:24:INFO:sockeye.training:__call__] E=38 B=154500	s/sec=2341.34 tok/sec=93221.69 u/sec=24.11	ppl=6.318797 
[2023-12-17:13:13:26:INFO:sockeye.training:__call__] E=38 B=154550	s/sec=1875.34 tok/sec=103384.22 u/sec=22.41	ppl=6.323128 
[2023-12-17:13:13:28:INFO:sockeye.training:__call__] E=38 B=154600	s/sec=1978.80 tok/sec=101516.52 u/sec=23.20	ppl=6.327103 
[2023-12-17:13:13:30:INFO:sockeye.training:__call__] E=38 B=154650	s/sec=2040.62 tok/sec=101750.87 u/sec=23.02	ppl=6.332145 
[2023-12-17:13:13:32:INFO:sockeye.training:__call__] E=38 B=154700	s/sec=2131.47 tok/sec=98517.74 u/sec=23.33	ppl=6.338938 
[2023-12-17:13:13:34:INFO:sockeye.training:__call__] E=38 B=154750	s/sec=2156.93 tok/sec=97153.10 u/sec=23.73	ppl=6.344830 
[2023-12-17:13:13:37:INFO:sockeye.training:__call__] E=38 B=154800	s/sec=2302.00 tok/sec=94210.86 u/sec=23.78	ppl=6.352431 
[2023-12-17:13:13:39:INFO:sockeye.training:__call__] E=38 B=154850	s/sec=2245.15 tok/sec=98029.42 u/sec=23.19	ppl=6.358686 
[2023-12-17:13:13:41:INFO:sockeye.training:__call__] E=38 B=154900	s/sec=2062.01 tok/sec=97150.38 u/sec=23.39	ppl=6.362779 
[2023-12-17:13:13:43:INFO:sockeye.training:__call__] E=38 B=154950	s/sec=1945.61 tok/sec=100221.20 u/sec=23.21	ppl=6.366304 
[2023-12-17:13:13:45:INFO:sockeye.training:__call__] E=38 B=155000	s/sec=1852.25 tok/sec=100720.91 u/sec=23.25	ppl=6.368048 
[2023-12-17:13:13:47:INFO:sockeye.training:__call__] E=38 B=155050	s/sec=2136.69 tok/sec=97075.64 u/sec=23.68	ppl=6.373438 
[2023-12-17:13:13:49:INFO:sockeye.training:__call__] E=38 B=155100	s/sec=2022.87 tok/sec=102048.00 u/sec=22.62	ppl=6.377889 
[2023-12-17:13:13:52:INFO:sockeye.training:__call__] E=38 B=155150	s/sec=1968.50 tok/sec=103861.69 u/sec=22.53	ppl=6.381181 
[2023-12-17:13:13:54:INFO:sockeye.training:__call__] E=38 B=155200	s/sec=1985.29 tok/sec=105872.77 u/sec=21.92	ppl=6.386023 
[2023-12-17:13:13:56:INFO:sockeye.training:__call__] E=38 B=155250	s/sec=2100.15 tok/sec=98610.64 u/sec=23.19	ppl=6.390888 
[2023-12-17:13:13:58:INFO:sockeye.training:__call__] E=38 B=155300	s/sec=1940.32 tok/sec=100679.87 u/sec=22.58	ppl=6.394376 
[2023-12-17:13:14:01:INFO:sockeye.training:__call__] E=38 B=155350	s/sec=1999.72 tok/sec=102491.37 u/sec=22.89	ppl=6.399221 
[2023-12-17:13:14:03:INFO:sockeye.training:__call__] E=38 B=155400	s/sec=2102.10 tok/sec=98498.33 u/sec=23.46	ppl=6.404183 
[2023-12-17:13:14:05:INFO:sockeye.training:__call__] E=38 B=155450	s/sec=2102.54 tok/sec=97847.47 u/sec=23.09	ppl=6.409239 
[2023-12-17:13:14:07:INFO:sockeye.training:__call__] E=38 B=155500	s/sec=2133.78 tok/sec=95298.87 u/sec=23.48	ppl=6.415632 
[2023-12-17:13:14:09:INFO:sockeye.training:__call__] E=38 B=155550	s/sec=2194.52 tok/sec=98637.68 u/sec=23.37	ppl=6.420776 
[2023-12-17:13:14:11:INFO:sockeye.training:__call__] E=38 B=155600	s/sec=2011.11 tok/sec=99358.40 u/sec=23.23	ppl=6.424887 
[2023-12-17:13:14:13:INFO:sockeye.training:__call__] E=38 B=155650	s/sec=1989.66 tok/sec=99639.35 u/sec=23.24	ppl=6.428570 
[2023-12-17:13:14:15:INFO:sockeye.training:__call__] E=38 B=155700	s/sec=2052.47 tok/sec=97339.29 u/sec=23.62	ppl=6.432443 
[2023-12-17:13:14:18:INFO:sockeye.training:__call__] E=38 B=155750	s/sec=2163.62 tok/sec=96440.01 u/sec=23.36	ppl=6.437901 
[2023-12-17:13:14:20:INFO:sockeye.training:__call__] E=38 B=155800	s/sec=1927.65 tok/sec=103448.33 u/sec=22.39	ppl=6.440087 
[2023-12-17:13:14:22:INFO:sockeye.training:__call__] E=39 B=155850	s/sec=1935.09 tok/sec=95100.21 u/sec=22.48	ppl=6.433869 
[2023-12-17:13:14:24:INFO:sockeye.training:__call__] E=39 B=155900	s/sec=2095.65 tok/sec=98748.10 u/sec=23.22	ppl=6.425990 
[2023-12-17:13:14:26:INFO:sockeye.training:__call__] E=39 B=155950	s/sec=1806.27 tok/sec=104262.06 u/sec=22.71	ppl=6.417547 
[2023-12-17:13:14:29:INFO:sockeye.training:__call__] E=39 B=156000	s/sec=1616.45 tok/sec=106734.52 u/sec=21.91	ppl=6.408377 
[2023-12-17:13:14:29:INFO:sockeye.training:_create_checkpoint] Checkpoint [39]	Updates=156000 Epoch=39 Samples=13901656 Time-cost=172.748 Updates/sec=23.155
[2023-12-17:13:14:29:INFO:sockeye.training:_create_checkpoint] Checkpoint [39]	Train-ppl=6.408377
[2023-12-17:13:14:31:INFO:sockeye.training:_evaluate] Checkpoint [39]	Validation-ppl=32.514162	Validation-bleu=0.171130	Validation-chrf=31.561320	Validation-rouge1=0.166971	Validation-rouge2=0.084283	Validation-rougel=0.160495	Validation-length-ratio-mse=1.275522	Validation-ter=95.130004	Validation-avg-sec-per-sent=0.003296	Validation-decode-walltime=1.647823
[2023-12-17:13:14:31:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 15 checkpoints, best so far: 32.780032
[2023-12-17:13:14:31:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:14:31:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [39]	Learning-rate=0.000146
[2023-12-17:13:14:32:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:14:32:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=7, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:14:32:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00039"
[2023-12-17:13:14:34:INFO:sockeye.training:__call__] E=39 B=156050	s/sec=864.75 tok/sec=38822.00 u/sec=9.08	ppl=5.916778 
[2023-12-17:13:14:36:INFO:sockeye.training:__call__] E=39 B=156100	s/sec=1968.52 tok/sec=101979.71 u/sec=23.04	ppl=5.904237 
[2023-12-17:13:14:39:INFO:sockeye.training:__call__] E=39 B=156150	s/sec=2111.70 tok/sec=98555.96 u/sec=23.32	ppl=5.878092 
[2023-12-17:13:14:41:INFO:sockeye.training:__call__] E=39 B=156200	s/sec=2149.66 tok/sec=97521.00 u/sec=23.28	ppl=5.897691 
[2023-12-17:13:14:43:INFO:sockeye.training:__call__] E=39 B=156250	s/sec=1874.90 tok/sec=99695.19 u/sec=23.39	ppl=5.887016 
[2023-12-17:13:14:45:INFO:sockeye.training:__call__] E=39 B=156300	s/sec=2074.53 tok/sec=99140.10 u/sec=22.79	ppl=5.888836 
[2023-12-17:13:14:47:INFO:sockeye.training:__call__] E=39 B=156350	s/sec=2202.49 tok/sec=97314.65 u/sec=23.45	ppl=5.915540 
[2023-12-17:13:14:49:INFO:sockeye.training:__call__] E=39 B=156400	s/sec=2276.11 tok/sec=96558.70 u/sec=23.67	ppl=5.936514 
[2023-12-17:13:14:51:INFO:sockeye.training:__call__] E=39 B=156450	s/sec=1866.09 tok/sec=100598.84 u/sec=23.10	ppl=5.935355 
[2023-12-17:13:14:54:INFO:sockeye.training:__call__] E=39 B=156500	s/sec=2269.97 tok/sec=97084.75 u/sec=23.72	ppl=5.955059 
[2023-12-17:13:14:56:INFO:sockeye.training:__call__] E=39 B=156550	s/sec=2149.45 tok/sec=98590.27 u/sec=23.40	ppl=5.968644 
[2023-12-17:13:14:58:INFO:sockeye.training:__call__] E=39 B=156600	s/sec=2045.39 tok/sec=100891.35 u/sec=22.79	ppl=5.978716 
[2023-12-17:13:15:00:INFO:sockeye.training:__call__] E=39 B=156650	s/sec=2068.80 tok/sec=100751.71 u/sec=22.64	ppl=5.988388 
[2023-12-17:13:15:02:INFO:sockeye.training:__call__] E=39 B=156700	s/sec=2063.64 tok/sec=100664.75 u/sec=22.87	ppl=5.990419 
[2023-12-17:13:15:05:INFO:sockeye.training:__call__] E=39 B=156750	s/sec=1760.72 tok/sec=104722.11 u/sec=22.14	ppl=5.993462 
[2023-12-17:13:15:07:INFO:sockeye.training:__call__] E=39 B=156800	s/sec=2155.58 tok/sec=98727.01 u/sec=23.15	ppl=6.000888 
[2023-12-17:13:15:09:INFO:sockeye.training:__call__] E=39 B=156850	s/sec=2222.23 tok/sec=98428.01 u/sec=23.07	ppl=6.010321 
[2023-12-17:13:15:11:INFO:sockeye.training:__call__] E=39 B=156900	s/sec=2163.79 tok/sec=96198.44 u/sec=23.48	ppl=6.023832 
[2023-12-17:13:15:13:INFO:sockeye.training:__call__] E=39 B=156950	s/sec=2064.10 tok/sec=99343.88 u/sec=23.20	ppl=6.024859 
[2023-12-17:13:15:15:INFO:sockeye.training:__call__] E=39 B=157000	s/sec=2068.53 tok/sec=96100.32 u/sec=23.51	ppl=6.030636 
[2023-12-17:13:15:17:INFO:sockeye.training:__call__] E=39 B=157050	s/sec=1993.69 tok/sec=100765.07 u/sec=23.29	ppl=6.037887 
[2023-12-17:13:15:20:INFO:sockeye.training:__call__] E=39 B=157100	s/sec=2246.77 tok/sec=95036.81 u/sec=23.88	ppl=6.046317 
[2023-12-17:13:15:22:INFO:sockeye.training:__call__] E=39 B=157150	s/sec=2048.94 tok/sec=101236.57 u/sec=22.79	ppl=6.051774 
[2023-12-17:13:15:24:INFO:sockeye.training:__call__] E=39 B=157200	s/sec=2157.87 tok/sec=97297.49 u/sec=23.70	ppl=6.061513 
[2023-12-17:13:15:26:INFO:sockeye.training:__call__] E=39 B=157250	s/sec=2086.63 tok/sec=97928.65 u/sec=23.54	ppl=6.071769 
[2023-12-17:13:15:28:INFO:sockeye.training:__call__] E=39 B=157300	s/sec=2191.14 tok/sec=98382.60 u/sec=23.17	ppl=6.078649 
[2023-12-17:13:15:30:INFO:sockeye.training:__call__] E=39 B=157350	s/sec=2206.43 tok/sec=96723.86 u/sec=23.37	ppl=6.086738 
[2023-12-17:13:15:32:INFO:sockeye.training:__call__] E=39 B=157400	s/sec=2247.04 tok/sec=98288.34 u/sec=23.29	ppl=6.094894 
[2023-12-17:13:15:35:INFO:sockeye.training:__call__] E=39 B=157450	s/sec=1692.57 tok/sec=104353.93 u/sec=22.32	ppl=6.099536 
[2023-12-17:13:15:37:INFO:sockeye.training:__call__] E=39 B=157500	s/sec=1813.96 tok/sec=101089.38 u/sec=22.63	ppl=6.102657 
[2023-12-17:13:15:39:INFO:sockeye.training:__call__] E=39 B=157550	s/sec=1981.91 tok/sec=101062.70 u/sec=22.94	ppl=6.105844 
[2023-12-17:13:15:41:INFO:sockeye.training:__call__] E=39 B=157600	s/sec=2418.55 tok/sec=93769.86 u/sec=23.69	ppl=6.118072 
[2023-12-17:13:15:43:INFO:sockeye.training:__call__] E=39 B=157650	s/sec=2124.90 tok/sec=98642.94 u/sec=23.18	ppl=6.126708 
[2023-12-17:13:15:45:INFO:sockeye.training:__call__] E=39 B=157700	s/sec=1916.91 tok/sec=98592.30 u/sec=23.26	ppl=6.131432 
[2023-12-17:13:15:48:INFO:sockeye.training:__call__] E=39 B=157750	s/sec=1963.13 tok/sec=101793.14 u/sec=22.64	ppl=6.134373 
[2023-12-17:13:15:50:INFO:sockeye.training:__call__] E=39 B=157800	s/sec=2103.97 tok/sec=98615.84 u/sec=23.19	ppl=6.143084 
[2023-12-17:13:15:52:INFO:sockeye.training:__call__] E=39 B=157850	s/sec=2003.24 tok/sec=98861.41 u/sec=23.49	ppl=6.146677 
[2023-12-17:13:15:54:INFO:sockeye.training:__call__] E=39 B=157900	s/sec=1996.60 tok/sec=98894.94 u/sec=23.19	ppl=6.150246 
[2023-12-17:13:15:56:INFO:sockeye.training:__call__] E=39 B=157950	s/sec=2141.02 tok/sec=94633.19 u/sec=23.90	ppl=6.156279 
[2023-12-17:13:15:58:INFO:sockeye.training:__call__] E=39 B=158000	s/sec=2200.02 tok/sec=96861.17 u/sec=23.63	ppl=6.162397 
[2023-12-17:13:16:00:INFO:sockeye.training:__call__] E=39 B=158050	s/sec=1999.09 tok/sec=99726.19 u/sec=23.01	ppl=6.168284 
[2023-12-17:13:16:03:INFO:sockeye.training:__call__] E=39 B=158100	s/sec=2169.63 tok/sec=99058.52 u/sec=23.50	ppl=6.174817 
[2023-12-17:13:16:05:INFO:sockeye.training:__call__] E=39 B=158150	s/sec=2338.67 tok/sec=92895.22 u/sec=24.24	ppl=6.182860 
[2023-12-17:13:16:07:INFO:sockeye.training:__call__] E=39 B=158200	s/sec=1760.59 tok/sec=102566.90 u/sec=22.83	ppl=6.184680 
[2023-12-17:13:16:09:INFO:sockeye.training:__call__] E=39 B=158250	s/sec=2175.40 tok/sec=94710.18 u/sec=23.85	ppl=6.191183 
[2023-12-17:13:16:11:INFO:sockeye.training:__call__] E=39 B=158300	s/sec=2157.39 tok/sec=97742.12 u/sec=23.29	ppl=6.197492 
[2023-12-17:13:16:13:INFO:sockeye.training:__call__] E=39 B=158350	s/sec=2064.05 tok/sec=100948.36 u/sec=22.87	ppl=6.201496 
[2023-12-17:13:16:15:INFO:sockeye.training:__call__] E=39 B=158400	s/sec=2270.90 tok/sec=95370.20 u/sec=23.93	ppl=6.210072 
[2023-12-17:13:16:18:INFO:sockeye.training:__call__] E=39 B=158450	s/sec=1888.83 tok/sec=104933.29 u/sec=22.66	ppl=6.213493 
[2023-12-17:13:16:20:INFO:sockeye.training:__call__] E=39 B=158500	s/sec=1975.52 tok/sec=102253.65 u/sec=22.41	ppl=6.218313 
[2023-12-17:13:16:22:INFO:sockeye.training:__call__] E=39 B=158550	s/sec=1911.36 tok/sec=99524.80 u/sec=23.29	ppl=6.221132 
[2023-12-17:13:16:24:INFO:sockeye.training:__call__] E=39 B=158600	s/sec=2172.72 tok/sec=106555.49 u/sec=22.12	ppl=6.226343 
[2023-12-17:13:16:26:INFO:sockeye.training:__call__] E=39 B=158650	s/sec=2221.64 tok/sec=98008.59 u/sec=23.57	ppl=6.231390 
[2023-12-17:13:16:29:INFO:sockeye.training:__call__] E=39 B=158700	s/sec=1882.90 tok/sec=104184.73 u/sec=22.33	ppl=6.235431 
[2023-12-17:13:16:31:INFO:sockeye.training:__call__] E=39 B=158750	s/sec=1967.57 tok/sec=97489.04 u/sec=23.56	ppl=6.240676 
[2023-12-17:13:16:33:INFO:sockeye.training:__call__] E=39 B=158800	s/sec=2008.44 tok/sec=98267.01 u/sec=23.46	ppl=6.245091 
[2023-12-17:13:16:35:INFO:sockeye.training:__call__] E=39 B=158850	s/sec=2174.48 tok/sec=97205.10 u/sec=23.64	ppl=6.250997 
[2023-12-17:13:16:37:INFO:sockeye.training:__call__] E=39 B=158900	s/sec=1955.77 tok/sec=101813.60 u/sec=22.22	ppl=6.254607 
[2023-12-17:13:16:39:INFO:sockeye.training:__call__] E=39 B=158950	s/sec=2117.09 tok/sec=100161.88 u/sec=23.25	ppl=6.257361 
[2023-12-17:13:16:42:INFO:sockeye.training:__call__] E=39 B=159000	s/sec=2039.82 tok/sec=98536.73 u/sec=23.26	ppl=6.260701 
[2023-12-17:13:16:44:INFO:sockeye.training:__call__] E=39 B=159050	s/sec=2363.68 tok/sec=96375.18 u/sec=23.41	ppl=6.266421 
[2023-12-17:13:16:46:INFO:sockeye.training:__call__] E=39 B=159100	s/sec=1834.66 tok/sec=101008.91 u/sec=22.93	ppl=6.267284 
[2023-12-17:13:16:48:INFO:sockeye.training:__call__] E=39 B=159150	s/sec=1964.05 tok/sec=102404.55 u/sec=22.52	ppl=6.271066 
[2023-12-17:13:16:50:INFO:sockeye.training:__call__] E=39 B=159200	s/sec=2334.38 tok/sec=93375.39 u/sec=24.24	ppl=6.277194 
[2023-12-17:13:16:52:INFO:sockeye.training:__call__] E=39 B=159250	s/sec=2151.66 tok/sec=94154.14 u/sec=23.84	ppl=6.281585 
[2023-12-17:13:16:54:INFO:sockeye.training:__call__] E=39 B=159300	s/sec=2230.20 tok/sec=94819.84 u/sec=23.83	ppl=6.287224 
[2023-12-17:13:16:56:INFO:sockeye.training:__call__] E=39 B=159350	s/sec=2360.01 tok/sec=93479.53 u/sec=23.98	ppl=6.294335 
[2023-12-17:13:16:59:INFO:sockeye.training:__call__] E=39 B=159400	s/sec=2014.94 tok/sec=100377.76 u/sec=23.06	ppl=6.298707 
[2023-12-17:13:17:01:INFO:sockeye.training:__call__] E=39 B=159450	s/sec=1840.83 tok/sec=101888.52 u/sec=22.83	ppl=6.300949 
[2023-12-17:13:17:03:INFO:sockeye.training:__call__] E=39 B=159500	s/sec=1851.51 tok/sec=103882.99 u/sec=22.13	ppl=6.303610 
[2023-12-17:13:17:05:INFO:sockeye.training:__call__] E=39 B=159550	s/sec=2105.49 tok/sec=98266.15 u/sec=23.50	ppl=6.308840 
[2023-12-17:13:17:07:INFO:sockeye.training:__call__] E=39 B=159600	s/sec=2240.96 tok/sec=95886.09 u/sec=23.58	ppl=6.314474 
[2023-12-17:13:17:09:INFO:sockeye.training:__call__] E=39 B=159650	s/sec=1963.69 tok/sec=102301.97 u/sec=22.85	ppl=6.319452 
[2023-12-17:13:17:12:INFO:sockeye.training:__call__] E=39 B=159700	s/sec=2197.19 tok/sec=96418.32 u/sec=23.72	ppl=6.324760 
[2023-12-17:13:17:14:INFO:sockeye.training:__call__] E=39 B=159750	s/sec=2003.25 tok/sec=99432.09 u/sec=23.06	ppl=6.328986 
[2023-12-17:13:17:16:INFO:sockeye.training:__call__] E=39 B=159800	s/sec=1907.37 tok/sec=99991.55 u/sec=22.97	ppl=6.333391 
[2023-12-17:13:17:18:INFO:sockeye.training:__call__] E=40 B=159850	s/sec=1965.74 tok/sec=97988.98 u/sec=21.67	ppl=6.325783 
[2023-12-17:13:17:20:INFO:sockeye.training:__call__] E=40 B=159900	s/sec=2200.04 tok/sec=98760.59 u/sec=23.54	ppl=6.318126 
[2023-12-17:13:17:22:INFO:sockeye.training:__call__] E=40 B=159950	s/sec=2025.14 tok/sec=98649.88 u/sec=23.01	ppl=6.310831 
[2023-12-17:13:17:25:INFO:sockeye.training:__call__] E=40 B=160000	s/sec=1914.37 tok/sec=100743.57 u/sec=23.37	ppl=6.302920 
[2023-12-17:13:17:25:INFO:sockeye.training:_create_checkpoint] Checkpoint [40]	Updates=160000 Epoch=40 Samples=14259304 Time-cost=172.559 Updates/sec=23.180
[2023-12-17:13:17:25:INFO:sockeye.training:_create_checkpoint] Checkpoint [40]	Train-ppl=6.302920
[2023-12-17:13:17:27:INFO:sockeye.training:_evaluate] Checkpoint [40]	Validation-ppl=32.755771	Validation-bleu=0.167581	Validation-chrf=31.524286	Validation-rouge1=0.170321	Validation-rouge2=0.081373	Validation-rougel=0.164012	Validation-length-ratio-mse=1.267890	Validation-ter=93.891870	Validation-avg-sec-per-sent=0.003265	Validation-decode-walltime=1.632673
[2023-12-17:13:17:27:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 16 checkpoints, best so far: 32.780032
[2023-12-17:13:17:27:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:17:27:INFO:sockeye.lr_scheduler:new_evaluation_result] 8 checkpoints since improvement or rate scaling, lowering learning rate: 1.46e-04 -> 1.31e-04
[2023-12-17:13:17:27:INFO:sockeye.training:_adjust_learning_rate] Loading model parameters and optimizer states from best checkpoint: 24
[2023-12-17:13:17:27:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best" to "cuda:0"
[2023-12-17:13:17:28:INFO:sockeye.training:_load_optimizer_state] Loaded optimizer state from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl"
[2023-12-17:13:17:28:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [40]	Learning-rate=0.000146
[2023-12-17:13:17:28:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:17:28:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:17:28:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00040"
[2023-12-17:13:17:30:INFO:sockeye.training:__call__] E=40 B=160050	s/sec=773.55 tok/sec=35344.69 u/sec=8.77	ppl=6.971254 
[2023-12-17:13:17:32:INFO:sockeye.training:__call__] E=40 B=160100	s/sec=2207.82 tok/sec=94940.00 u/sec=23.75	ppl=6.981327 
[2023-12-17:13:17:35:INFO:sockeye.training:__call__] E=40 B=160150	s/sec=1910.19 tok/sec=103466.22 u/sec=22.44	ppl=6.966730 
[2023-12-17:13:17:37:INFO:sockeye.training:__call__] E=40 B=160200	s/sec=2174.92 tok/sec=96757.22 u/sec=23.68	ppl=6.941122 
[2023-12-17:13:17:39:INFO:sockeye.training:__call__] E=40 B=160250	s/sec=2009.95 tok/sec=100278.43 u/sec=22.84	ppl=6.930059 
[2023-12-17:13:17:41:INFO:sockeye.training:__call__] E=40 B=160300	s/sec=2148.34 tok/sec=97892.35 u/sec=23.47	ppl=6.934282 
[2023-12-17:13:17:43:INFO:sockeye.training:__call__] E=40 B=160350	s/sec=1831.88 tok/sec=103978.14 u/sec=22.67	ppl=6.911379 
[2023-12-17:13:17:45:INFO:sockeye.training:__call__] E=40 B=160400	s/sec=2165.03 tok/sec=99103.91 u/sec=23.09	ppl=6.913961 
[2023-12-17:13:17:48:INFO:sockeye.training:__call__] E=40 B=160450	s/sec=1945.35 tok/sec=102783.75 u/sec=22.47	ppl=6.920765 
[2023-12-17:13:17:50:INFO:sockeye.training:__call__] E=40 B=160500	s/sec=2228.75 tok/sec=95517.67 u/sec=23.69	ppl=6.942921 
[2023-12-17:13:17:52:INFO:sockeye.training:__call__] E=40 B=160550	s/sec=1780.81 tok/sec=101354.62 u/sec=23.14	ppl=6.946847 
[2023-12-17:13:17:54:INFO:sockeye.training:__call__] E=40 B=160600	s/sec=2086.25 tok/sec=99452.07 u/sec=23.37	ppl=6.950767 
[2023-12-17:13:17:56:INFO:sockeye.training:__call__] E=40 B=160650	s/sec=1741.81 tok/sec=103205.55 u/sec=22.13	ppl=6.953201 
[2023-12-17:13:17:59:INFO:sockeye.training:__call__] E=40 B=160700	s/sec=2014.42 tok/sec=100829.63 u/sec=22.60	ppl=6.954211 
[2023-12-17:13:18:01:INFO:sockeye.training:__call__] E=40 B=160750	s/sec=2010.68 tok/sec=101138.25 u/sec=23.31	ppl=6.956891 
[2023-12-17:13:18:03:INFO:sockeye.training:__call__] E=40 B=160800	s/sec=2190.23 tok/sec=97592.10 u/sec=23.28	ppl=6.966717 
[2023-12-17:13:18:05:INFO:sockeye.training:__call__] E=40 B=160850	s/sec=2045.51 tok/sec=99973.01 u/sec=23.29	ppl=6.975635 
[2023-12-17:13:18:07:INFO:sockeye.training:__call__] E=40 B=160900	s/sec=2274.69 tok/sec=97093.06 u/sec=23.27	ppl=6.988377 
[2023-12-17:13:18:09:INFO:sockeye.training:__call__] E=40 B=160950	s/sec=2088.73 tok/sec=97374.73 u/sec=23.23	ppl=7.001802 
[2023-12-17:13:18:11:INFO:sockeye.training:__call__] E=40 B=161000	s/sec=2165.77 tok/sec=95278.74 u/sec=23.71	ppl=7.010903 
[2023-12-17:13:18:14:INFO:sockeye.training:__call__] E=40 B=161050	s/sec=2035.28 tok/sec=98140.59 u/sec=23.30	ppl=7.017215 
[2023-12-17:13:18:16:INFO:sockeye.training:__call__] E=40 B=161100	s/sec=2217.71 tok/sec=97579.18 u/sec=23.41	ppl=7.023149 
[2023-12-17:13:18:18:INFO:sockeye.training:__call__] E=40 B=161150	s/sec=2001.98 tok/sec=97228.79 u/sec=22.71	ppl=7.034241 
[2023-12-17:13:18:20:INFO:sockeye.training:__call__] E=40 B=161200	s/sec=2152.71 tok/sec=98280.66 u/sec=23.48	ppl=7.036585 
[2023-12-17:13:18:22:INFO:sockeye.training:__call__] E=40 B=161250	s/sec=2190.38 tok/sec=96301.09 u/sec=23.64	ppl=7.048106 
[2023-12-17:13:18:24:INFO:sockeye.training:__call__] E=40 B=161300	s/sec=1984.82 tok/sec=100397.27 u/sec=23.02	ppl=7.052226 
[2023-12-17:13:18:27:INFO:sockeye.training:__call__] E=40 B=161350	s/sec=1989.03 tok/sec=100090.39 u/sec=22.69	ppl=7.060607 
[2023-12-17:13:18:29:INFO:sockeye.training:__call__] E=40 B=161400	s/sec=2021.31 tok/sec=100180.16 u/sec=23.05	ppl=7.065776 
[2023-12-17:13:18:31:INFO:sockeye.training:__call__] E=40 B=161450	s/sec=2134.09 tok/sec=96895.77 u/sec=23.52	ppl=7.076031 
[2023-12-17:13:18:33:INFO:sockeye.training:__call__] E=40 B=161500	s/sec=2105.61 tok/sec=95363.37 u/sec=23.71	ppl=7.081216 
[2023-12-17:13:18:35:INFO:sockeye.training:__call__] E=40 B=161550	s/sec=2119.18 tok/sec=99797.31 u/sec=23.11	ppl=7.088498 
[2023-12-17:13:18:37:INFO:sockeye.training:__call__] E=40 B=161600	s/sec=2143.20 tok/sec=97699.90 u/sec=23.50	ppl=7.098004 
[2023-12-17:13:18:39:INFO:sockeye.training:__call__] E=40 B=161650	s/sec=2006.88 tok/sec=100146.11 u/sec=22.93	ppl=7.098984 
[2023-12-17:13:18:42:INFO:sockeye.training:__call__] E=40 B=161700	s/sec=2158.36 tok/sec=97303.42 u/sec=23.10	ppl=7.107818 
[2023-12-17:13:18:44:INFO:sockeye.training:__call__] E=40 B=161750	s/sec=2407.15 tok/sec=94234.89 u/sec=23.92	ppl=7.114989 
[2023-12-17:13:18:46:INFO:sockeye.training:__call__] E=40 B=161800	s/sec=2055.75 tok/sec=97799.41 u/sec=23.62	ppl=7.121218 
[2023-12-17:13:18:48:INFO:sockeye.training:__call__] E=40 B=161850	s/sec=2315.43 tok/sec=95168.91 u/sec=23.45	ppl=7.129881 
[2023-12-17:13:18:50:INFO:sockeye.training:__call__] E=40 B=161900	s/sec=2191.11 tok/sec=97976.48 u/sec=23.33	ppl=7.132946 
[2023-12-17:13:18:52:INFO:sockeye.training:__call__] E=40 B=161950	s/sec=1964.57 tok/sec=99418.93 u/sec=23.25	ppl=7.136670 
[2023-12-17:13:18:54:INFO:sockeye.training:__call__] E=40 B=162000	s/sec=1921.52 tok/sec=100740.27 u/sec=22.92	ppl=7.141824 
[2023-12-17:13:18:57:INFO:sockeye.training:__call__] E=40 B=162050	s/sec=1890.48 tok/sec=102882.46 u/sec=22.42	ppl=7.145756 
[2023-12-17:13:18:59:INFO:sockeye.training:__call__] E=40 B=162100	s/sec=2196.95 tok/sec=96332.55 u/sec=23.67	ppl=7.151042 
[2023-12-17:13:19:01:INFO:sockeye.training:__call__] E=40 B=162150	s/sec=2101.64 tok/sec=98489.67 u/sec=23.33	ppl=7.156716 
[2023-12-17:13:19:03:INFO:sockeye.training:__call__] E=40 B=162200	s/sec=2369.22 tok/sec=94160.39 u/sec=23.77	ppl=7.166190 
[2023-12-17:13:19:05:INFO:sockeye.training:__call__] E=40 B=162250	s/sec=2001.14 tok/sec=102727.37 u/sec=22.86	ppl=7.171398 
[2023-12-17:13:19:07:INFO:sockeye.training:__call__] E=40 B=162300	s/sec=2149.98 tok/sec=97159.91 u/sec=23.13	ppl=7.176545 
[2023-12-17:13:19:10:INFO:sockeye.training:__call__] E=40 B=162350	s/sec=1833.47 tok/sec=101891.42 u/sec=22.65	ppl=7.181568 
[2023-12-17:13:19:12:INFO:sockeye.training:__call__] E=40 B=162400	s/sec=2063.31 tok/sec=100373.44 u/sec=22.66	ppl=7.187084 
[2023-12-17:13:19:14:INFO:sockeye.training:__call__] E=40 B=162450	s/sec=2138.60 tok/sec=99099.55 u/sec=23.33	ppl=7.195262 
[2023-12-17:13:19:16:INFO:sockeye.training:__call__] E=40 B=162500	s/sec=2220.74 tok/sec=96994.88 u/sec=23.37	ppl=7.203703 
[2023-12-17:13:19:18:INFO:sockeye.training:__call__] E=40 B=162550	s/sec=1887.80 tok/sec=102790.17 u/sec=22.87	ppl=7.207822 
[2023-12-17:13:19:20:INFO:sockeye.training:__call__] E=40 B=162600	s/sec=2013.97 tok/sec=99494.49 u/sec=23.22	ppl=7.211508 
[2023-12-17:13:19:23:INFO:sockeye.training:__call__] E=40 B=162650	s/sec=1888.06 tok/sec=101799.87 u/sec=23.09	ppl=7.214044 
[2023-12-17:13:19:25:INFO:sockeye.training:__call__] E=40 B=162700	s/sec=2219.67 tok/sec=95558.55 u/sec=23.51	ppl=7.223162 
[2023-12-17:13:19:27:INFO:sockeye.training:__call__] E=40 B=162750	s/sec=2134.23 tok/sec=98493.56 u/sec=23.20	ppl=7.229979 
[2023-12-17:13:19:29:INFO:sockeye.training:__call__] E=40 B=162800	s/sec=2019.89 tok/sec=98805.65 u/sec=22.95	ppl=7.236394 
[2023-12-17:13:19:31:INFO:sockeye.training:__call__] E=40 B=162850	s/sec=2136.69 tok/sec=98699.44 u/sec=23.39	ppl=7.242859 
[2023-12-17:13:19:33:INFO:sockeye.training:__call__] E=40 B=162900	s/sec=2061.95 tok/sec=97691.02 u/sec=23.30	ppl=7.251199 
[2023-12-17:13:19:35:INFO:sockeye.training:__call__] E=40 B=162950	s/sec=2135.69 tok/sec=98010.53 u/sec=23.30	ppl=7.254845 
[2023-12-17:13:19:38:INFO:sockeye.training:__call__] E=40 B=163000	s/sec=2311.24 tok/sec=94069.51 u/sec=23.88	ppl=7.261404 
[2023-12-17:13:19:40:INFO:sockeye.training:__call__] E=40 B=163050	s/sec=1854.42 tok/sec=103553.85 u/sec=22.95	ppl=7.264887 
[2023-12-17:13:19:42:INFO:sockeye.training:__call__] E=40 B=163100	s/sec=1914.87 tok/sec=100408.14 u/sec=22.88	ppl=7.269861 
[2023-12-17:13:19:44:INFO:sockeye.training:__call__] E=40 B=163150	s/sec=2007.44 tok/sec=108920.87 u/sec=21.34	ppl=7.273861 
[2023-12-17:13:19:46:INFO:sockeye.training:__call__] E=40 B=163200	s/sec=1820.51 tok/sec=103905.93 u/sec=22.49	ppl=7.275406 
[2023-12-17:13:19:49:INFO:sockeye.training:__call__] E=40 B=163250	s/sec=2126.73 tok/sec=96492.11 u/sec=23.48	ppl=7.281839 
[2023-12-17:13:19:51:INFO:sockeye.training:__call__] E=40 B=163300	s/sec=1951.22 tok/sec=98622.50 u/sec=23.36	ppl=7.283553 
[2023-12-17:13:19:53:INFO:sockeye.training:__call__] E=40 B=163350	s/sec=2130.55 tok/sec=95803.24 u/sec=23.48	ppl=7.290226 
[2023-12-17:13:19:55:INFO:sockeye.training:__call__] E=40 B=163400	s/sec=2243.24 tok/sec=94966.55 u/sec=23.93	ppl=7.296059 
[2023-12-17:13:19:57:INFO:sockeye.training:__call__] E=40 B=163450	s/sec=2068.14 tok/sec=99091.23 u/sec=23.37	ppl=7.300733 
[2023-12-17:13:19:59:INFO:sockeye.training:__call__] E=40 B=163500	s/sec=1951.46 tok/sec=102772.28 u/sec=22.50	ppl=7.304328 
[2023-12-17:13:20:01:INFO:sockeye.training:__call__] E=40 B=163550	s/sec=2091.76 tok/sec=99980.12 u/sec=23.68	ppl=7.307709 
[2023-12-17:13:20:04:INFO:sockeye.training:__call__] E=40 B=163600	s/sec=2041.26 tok/sec=99467.65 u/sec=23.24	ppl=7.309578 
[2023-12-17:13:20:06:INFO:sockeye.training:__call__] E=40 B=163650	s/sec=2030.77 tok/sec=101583.28 u/sec=23.25	ppl=7.313850 
[2023-12-17:13:20:08:INFO:sockeye.training:__call__] E=40 B=163700	s/sec=2052.27 tok/sec=101135.71 u/sec=23.32	ppl=7.317833 
[2023-12-17:13:20:10:INFO:sockeye.training:__call__] E=40 B=163750	s/sec=2178.15 tok/sec=99840.03 u/sec=23.59	ppl=7.322287 
[2023-12-17:13:20:12:INFO:sockeye.training:__call__] E=41 B=163800	s/sec=1730.00 tok/sec=97159.92 u/sec=21.89	ppl=7.324189 
[2023-12-17:13:20:14:INFO:sockeye.training:__call__] E=41 B=163850	s/sec=2178.00 tok/sec=97289.27 u/sec=23.72	ppl=7.314700 
[2023-12-17:13:20:17:INFO:sockeye.training:__call__] E=41 B=163900	s/sec=2053.15 tok/sec=99417.17 u/sec=23.33	ppl=7.304136 
[2023-12-17:13:20:19:INFO:sockeye.training:__call__] E=41 B=163950	s/sec=2177.63 tok/sec=100341.54 u/sec=23.19	ppl=7.295217 
[2023-12-17:13:20:21:INFO:sockeye.training:__call__] E=41 B=164000	s/sec=2408.59 tok/sec=94127.58 u/sec=24.05	ppl=7.287471 
[2023-12-17:13:20:21:INFO:sockeye.training:_create_checkpoint] Checkpoint [41]	Updates=164000 Epoch=41 Samples=14616912 Time-cost=172.533 Updates/sec=23.184
[2023-12-17:13:20:21:INFO:sockeye.training:_create_checkpoint] Checkpoint [41]	Train-ppl=7.287471
[2023-12-17:13:20:23:INFO:sockeye.training:_evaluate] Checkpoint [41]	Validation-ppl=28.929884	Validation-bleu=0.168571	Validation-chrf=31.142122	Validation-rouge1=0.164500	Validation-rouge2=0.079600	Validation-rougel=0.157009	Validation-length-ratio-mse=1.270861	Validation-ter=94.510937	Validation-avg-sec-per-sent=0.003281	Validation-decode-walltime=1.640489
[2023-12-17:13:20:23:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 17 checkpoints, best so far: 32.780032
[2023-12-17:13:20:23:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:20:23:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [41]	Learning-rate=0.000131
[2023-12-17:13:20:24:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:20:24:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:20:24:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00041"
[2023-12-17:13:20:26:INFO:sockeye.training:__call__] E=41 B=164050	s/sec=786.43 tok/sec=37815.47 u/sec=8.73	ppl=6.683020 
[2023-12-17:13:20:29:INFO:sockeye.training:__call__] E=41 B=164100	s/sec=2115.07 tok/sec=99613.97 u/sec=22.95	ppl=6.706148 
[2023-12-17:13:20:31:INFO:sockeye.training:__call__] E=41 B=164150	s/sec=2103.79 tok/sec=101857.03 u/sec=22.79	ppl=6.657124 
[2023-12-17:13:20:33:INFO:sockeye.training:__call__] E=41 B=164200	s/sec=2034.85 tok/sec=100147.96 u/sec=22.75	ppl=6.654903 
[2023-12-17:13:20:35:INFO:sockeye.training:__call__] E=41 B=164250	s/sec=1899.13 tok/sec=100040.97 u/sec=23.37	ppl=6.656226 
[2023-12-17:13:20:37:INFO:sockeye.training:__call__] E=41 B=164300	s/sec=2207.23 tok/sec=97923.50 u/sec=23.30	ppl=6.679209 
[2023-12-17:13:20:40:INFO:sockeye.training:__call__] E=41 B=164350	s/sec=1877.89 tok/sec=103524.76 u/sec=22.44	ppl=6.672182 
[2023-12-17:13:20:42:INFO:sockeye.training:__call__] E=41 B=164400	s/sec=2031.13 tok/sec=101665.35 u/sec=22.63	ppl=6.677978 
[2023-12-17:13:20:44:INFO:sockeye.training:__call__] E=41 B=164450	s/sec=2023.62 tok/sec=95843.74 u/sec=24.00	ppl=6.675015 
[2023-12-17:13:20:46:INFO:sockeye.training:__call__] E=41 B=164500	s/sec=2006.88 tok/sec=98533.22 u/sec=23.58	ppl=6.683593 
[2023-12-17:13:20:48:INFO:sockeye.training:__call__] E=41 B=164550	s/sec=2259.67 tok/sec=97874.29 u/sec=23.31	ppl=6.698227 
[2023-12-17:13:20:50:INFO:sockeye.training:__call__] E=41 B=164600	s/sec=1984.33 tok/sec=100824.46 u/sec=23.05	ppl=6.703074 
[2023-12-17:13:20:52:INFO:sockeye.training:__call__] E=41 B=164650	s/sec=2075.48 tok/sec=96526.72 u/sec=23.71	ppl=6.706315 
[2023-12-17:13:20:55:INFO:sockeye.training:__call__] E=41 B=164700	s/sec=1798.16 tok/sec=101197.25 u/sec=22.98	ppl=6.710906 
[2023-12-17:13:20:57:INFO:sockeye.training:__call__] E=41 B=164750	s/sec=2026.42 tok/sec=97146.42 u/sec=23.76	ppl=6.718350 
[2023-12-17:13:20:59:INFO:sockeye.training:__call__] E=41 B=164800	s/sec=2131.37 tok/sec=98937.51 u/sec=23.49	ppl=6.730959 
[2023-12-17:13:21:01:INFO:sockeye.training:__call__] E=41 B=164850	s/sec=2069.38 tok/sec=102044.47 u/sec=23.30	ppl=6.736213 
[2023-12-17:13:21:03:INFO:sockeye.training:__call__] E=41 B=164900	s/sec=2032.46 tok/sec=99208.03 u/sec=23.44	ppl=6.744864 
[2023-12-17:13:21:05:INFO:sockeye.training:__call__] E=41 B=164950	s/sec=1820.59 tok/sec=107428.82 u/sec=22.05	ppl=6.742694 
[2023-12-17:13:21:07:INFO:sockeye.training:__call__] E=41 B=165000	s/sec=2309.83 tok/sec=97339.15 u/sec=23.71	ppl=6.755218 
[2023-12-17:13:21:10:INFO:sockeye.training:__call__] E=41 B=165050	s/sec=2057.04 tok/sec=101510.66 u/sec=23.29	ppl=6.767109 
[2023-12-17:13:21:12:INFO:sockeye.training:__call__] E=41 B=165100	s/sec=2017.93 tok/sec=102877.69 u/sec=22.93	ppl=6.775086 
[2023-12-17:13:21:14:INFO:sockeye.training:__call__] E=41 B=165150	s/sec=1833.64 tok/sec=101873.80 u/sec=23.44	ppl=6.780168 
[2023-12-17:13:21:16:INFO:sockeye.training:__call__] E=41 B=165200	s/sec=2053.37 tok/sec=101290.17 u/sec=22.20	ppl=6.789635 
[2023-12-17:13:21:18:INFO:sockeye.training:__call__] E=41 B=165250	s/sec=2080.60 tok/sec=98940.91 u/sec=23.39	ppl=6.797189 
[2023-12-17:13:21:20:INFO:sockeye.training:__call__] E=41 B=165300	s/sec=2095.23 tok/sec=95869.34 u/sec=23.85	ppl=6.810973 
[2023-12-17:13:21:23:INFO:sockeye.training:__call__] E=41 B=165350	s/sec=1894.93 tok/sec=102326.26 u/sec=23.13	ppl=6.812566 
[2023-12-17:13:21:25:INFO:sockeye.training:__call__] E=41 B=165400	s/sec=2083.10 tok/sec=98913.46 u/sec=23.33	ppl=6.822780 
[2023-12-17:13:21:27:INFO:sockeye.training:__call__] E=41 B=165450	s/sec=2034.79 tok/sec=100319.10 u/sec=23.00	ppl=6.825690 
[2023-12-17:13:21:29:INFO:sockeye.training:__call__] E=41 B=165500	s/sec=1864.70 tok/sec=99671.08 u/sec=23.50	ppl=6.828389 
[2023-12-17:13:21:31:INFO:sockeye.training:__call__] E=41 B=165550	s/sec=2229.78 tok/sec=99044.27 u/sec=23.54	ppl=6.838280 
[2023-12-17:13:21:33:INFO:sockeye.training:__call__] E=41 B=165600	s/sec=2141.26 tok/sec=99306.70 u/sec=23.40	ppl=6.844398 
[2023-12-17:13:21:35:INFO:sockeye.training:__call__] E=41 B=165650	s/sec=2000.08 tok/sec=101347.16 u/sec=23.19	ppl=6.849517 
[2023-12-17:13:21:38:INFO:sockeye.training:__call__] E=41 B=165700	s/sec=2155.90 tok/sec=97653.48 u/sec=23.60	ppl=6.858972 
[2023-12-17:13:21:40:INFO:sockeye.training:__call__] E=41 B=165750	s/sec=2224.76 tok/sec=97312.31 u/sec=23.73	ppl=6.865616 
[2023-12-17:13:21:42:INFO:sockeye.training:__call__] E=41 B=165800	s/sec=1949.51 tok/sec=101890.56 u/sec=23.03	ppl=6.869193 
[2023-12-17:13:21:44:INFO:sockeye.training:__call__] E=41 B=165850	s/sec=2159.96 tok/sec=97828.07 u/sec=23.44	ppl=6.878470 
[2023-12-17:13:21:46:INFO:sockeye.training:__call__] E=41 B=165900	s/sec=2056.32 tok/sec=97673.41 u/sec=23.67	ppl=6.884696 
[2023-12-17:13:21:48:INFO:sockeye.training:__call__] E=41 B=165950	s/sec=1885.86 tok/sec=101684.07 u/sec=22.71	ppl=6.888721 
[2023-12-17:13:21:50:INFO:sockeye.training:__call__] E=41 B=166000	s/sec=2212.85 tok/sec=97111.73 u/sec=23.32	ppl=6.893977 
[2023-12-17:13:21:53:INFO:sockeye.training:__call__] E=41 B=166050	s/sec=2213.66 tok/sec=96976.65 u/sec=23.69	ppl=6.901399 
[2023-12-17:13:21:55:INFO:sockeye.training:__call__] E=41 B=166100	s/sec=2304.90 tok/sec=96205.93 u/sec=23.73	ppl=6.906341 
[2023-12-17:13:21:57:INFO:sockeye.training:__call__] E=41 B=166150	s/sec=2034.99 tok/sec=99481.74 u/sec=23.47	ppl=6.913572 
[2023-12-17:13:21:59:INFO:sockeye.training:__call__] E=41 B=166200	s/sec=2148.66 tok/sec=97555.32 u/sec=23.31	ppl=6.919218 
[2023-12-17:13:22:01:INFO:sockeye.training:__call__] E=41 B=166250	s/sec=1961.93 tok/sec=101743.61 u/sec=22.75	ppl=6.925808 
[2023-12-17:13:22:03:INFO:sockeye.training:__call__] E=41 B=166300	s/sec=2146.13 tok/sec=99412.25 u/sec=23.45	ppl=6.935134 
[2023-12-17:13:22:06:INFO:sockeye.training:__call__] E=41 B=166350	s/sec=1760.60 tok/sec=104474.26 u/sec=22.41	ppl=6.939619 
[2023-12-17:13:22:08:INFO:sockeye.training:__call__] E=41 B=166400	s/sec=2079.41 tok/sec=98909.19 u/sec=23.50	ppl=6.947509 
[2023-12-17:13:22:10:INFO:sockeye.training:__call__] E=41 B=166450	s/sec=2038.09 tok/sec=99870.03 u/sec=23.46	ppl=6.953784 
[2023-12-17:13:22:12:INFO:sockeye.training:__call__] E=41 B=166500	s/sec=2276.59 tok/sec=95891.80 u/sec=23.87	ppl=6.960770 
[2023-12-17:13:22:14:INFO:sockeye.training:__call__] E=41 B=166550	s/sec=1919.47 tok/sec=103759.82 u/sec=22.59	ppl=6.963835 
[2023-12-17:13:22:16:INFO:sockeye.training:__call__] E=41 B=166600	s/sec=2205.39 tok/sec=98565.21 u/sec=23.64	ppl=6.970113 
[2023-12-17:13:22:18:INFO:sockeye.training:__call__] E=41 B=166650	s/sec=2210.61 tok/sec=96916.62 u/sec=23.78	ppl=6.976808 
[2023-12-17:13:22:20:INFO:sockeye.training:__call__] E=41 B=166700	s/sec=1945.71 tok/sec=102351.89 u/sec=22.94	ppl=6.982677 
[2023-12-17:13:22:23:INFO:sockeye.training:__call__] E=41 B=166750	s/sec=2173.45 tok/sec=97419.22 u/sec=23.67	ppl=6.990032 
[2023-12-17:13:22:25:INFO:sockeye.training:__call__] E=41 B=166800	s/sec=2268.93 tok/sec=95983.07 u/sec=24.16	ppl=6.997515 
[2023-12-17:13:22:27:INFO:sockeye.training:__call__] E=41 B=166850	s/sec=2269.74 tok/sec=97865.88 u/sec=23.53	ppl=7.003375 
[2023-12-17:13:22:29:INFO:sockeye.training:__call__] E=41 B=166900	s/sec=2216.48 tok/sec=95165.84 u/sec=24.09	ppl=7.010440 
[2023-12-17:13:22:31:INFO:sockeye.training:__call__] E=41 B=166950	s/sec=1887.08 tok/sec=102249.41 u/sec=23.08	ppl=7.015408 
[2023-12-17:13:22:33:INFO:sockeye.training:__call__] E=41 B=167000	s/sec=1966.47 tok/sec=102184.71 u/sec=22.47	ppl=7.019851 
[2023-12-17:13:22:35:INFO:sockeye.training:__call__] E=41 B=167050	s/sec=1923.05 tok/sec=103837.02 u/sec=23.16	ppl=7.024492 
[2023-12-17:13:22:38:INFO:sockeye.training:__call__] E=41 B=167100	s/sec=1979.90 tok/sec=100058.74 u/sec=23.09	ppl=7.028457 
[2023-12-17:13:22:40:INFO:sockeye.training:__call__] E=41 B=167150	s/sec=2215.04 tok/sec=97165.21 u/sec=23.15	ppl=7.034305 
[2023-12-17:13:22:42:INFO:sockeye.training:__call__] E=41 B=167200	s/sec=2094.36 tok/sec=107068.38 u/sec=21.93	ppl=7.039100 
[2023-12-17:13:22:44:INFO:sockeye.training:__call__] E=41 B=167250	s/sec=2091.69 tok/sec=98926.63 u/sec=23.51	ppl=7.043964 
[2023-12-17:13:22:46:INFO:sockeye.training:__call__] E=41 B=167300	s/sec=1993.52 tok/sec=100207.52 u/sec=23.07	ppl=7.048765 
[2023-12-17:13:22:48:INFO:sockeye.training:__call__] E=41 B=167350	s/sec=2047.06 tok/sec=98198.41 u/sec=23.61	ppl=7.054390 
[2023-12-17:13:22:51:INFO:sockeye.training:__call__] E=41 B=167400	s/sec=2215.82 tok/sec=98187.27 u/sec=23.47	ppl=7.061796 
[2023-12-17:13:22:53:INFO:sockeye.training:__call__] E=41 B=167450	s/sec=2158.60 tok/sec=97995.94 u/sec=23.34	ppl=7.068126 
[2023-12-17:13:22:55:INFO:sockeye.training:__call__] E=41 B=167500	s/sec=1980.12 tok/sec=101734.06 u/sec=22.92	ppl=7.073173 
[2023-12-17:13:22:57:INFO:sockeye.training:__call__] E=41 B=167550	s/sec=1866.40 tok/sec=102891.42 u/sec=22.69	ppl=7.076700 
[2023-12-17:13:22:59:INFO:sockeye.training:__call__] E=41 B=167600	s/sec=1964.75 tok/sec=102357.97 u/sec=22.95	ppl=7.080580 
[2023-12-17:13:23:01:INFO:sockeye.training:__call__] E=41 B=167650	s/sec=2089.63 tok/sec=100420.60 u/sec=23.07	ppl=7.085158 
[2023-12-17:13:23:04:INFO:sockeye.training:__call__] E=41 B=167700	s/sec=2203.77 tok/sec=95097.97 u/sec=24.12	ppl=7.089622 
[2023-12-17:13:23:06:INFO:sockeye.training:__call__] E=41 B=167750	s/sec=2340.24 tok/sec=92960.66 u/sec=24.14	ppl=7.096716 
[2023-12-17:13:23:08:INFO:sockeye.training:__call__] E=42 B=167800	s/sec=1911.48 tok/sec=95953.81 u/sec=21.18	ppl=7.098501 
[2023-12-17:13:23:10:INFO:sockeye.training:__call__] E=42 B=167850	s/sec=2044.15 tok/sec=100621.32 u/sec=23.27	ppl=7.089217 
[2023-12-17:13:23:12:INFO:sockeye.training:__call__] E=42 B=167900	s/sec=2290.87 tok/sec=94119.81 u/sec=24.27	ppl=7.079921 
[2023-12-17:13:23:14:INFO:sockeye.training:__call__] E=42 B=167950	s/sec=1943.10 tok/sec=103195.86 u/sec=23.22	ppl=7.069359 
[2023-12-17:13:23:16:INFO:sockeye.training:__call__] E=42 B=168000	s/sec=2246.89 tok/sec=97720.98 u/sec=23.48	ppl=7.061089 
[2023-12-17:13:23:16:INFO:sockeye.training:_create_checkpoint] Checkpoint [42]	Updates=168000 Epoch=42 Samples=14972672 Time-cost=172.099 Updates/sec=23.242
[2023-12-17:13:23:16:INFO:sockeye.training:_create_checkpoint] Checkpoint [42]	Train-ppl=7.061089
[2023-12-17:13:23:19:INFO:sockeye.training:_evaluate] Checkpoint [42]	Validation-ppl=30.015921	Validation-bleu=0.153314	Validation-chrf=29.776083	Validation-rouge1=0.152118	Validation-rouge2=0.074443	Validation-rougel=0.145181	Validation-length-ratio-mse=1.337104	Validation-ter=100.701610	Validation-avg-sec-per-sent=0.003569	Validation-decode-walltime=1.784608
[2023-12-17:13:23:19:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 18 checkpoints, best so far: 32.780032
[2023-12-17:13:23:19:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:23:19:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [42]	Learning-rate=0.000131
[2023-12-17:13:23:20:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:23:20:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:23:20:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00042"
[2023-12-17:13:23:22:INFO:sockeye.training:__call__] E=42 B=168050	s/sec=668.74 tok/sec=39315.51 u/sec=8.43	ppl=6.357637 
[2023-12-17:13:23:24:INFO:sockeye.training:__call__] E=42 B=168100	s/sec=2231.16 tok/sec=97332.51 u/sec=23.40	ppl=6.432803 
[2023-12-17:13:23:27:INFO:sockeye.training:__call__] E=42 B=168150	s/sec=1985.15 tok/sec=101880.04 u/sec=22.89	ppl=6.427351 
[2023-12-17:13:23:29:INFO:sockeye.training:__call__] E=42 B=168200	s/sec=2039.51 tok/sec=98166.32 u/sec=23.43	ppl=6.422682 
[2023-12-17:13:23:31:INFO:sockeye.training:__call__] E=42 B=168250	s/sec=2068.92 tok/sec=98610.92 u/sec=23.68	ppl=6.433397 
[2023-12-17:13:23:33:INFO:sockeye.training:__call__] E=42 B=168300	s/sec=1918.01 tok/sec=100785.43 u/sec=23.41	ppl=6.443893 
[2023-12-17:13:23:35:INFO:sockeye.training:__call__] E=42 B=168350	s/sec=2106.67 tok/sec=98311.43 u/sec=23.60	ppl=6.443689 
[2023-12-17:13:23:37:INFO:sockeye.training:__call__] E=42 B=168400	s/sec=2100.42 tok/sec=99335.87 u/sec=23.19	ppl=6.453297 
[2023-12-17:13:23:40:INFO:sockeye.training:__call__] E=42 B=168450	s/sec=1948.79 tok/sec=100760.00 u/sec=23.11	ppl=6.450913 
[2023-12-17:13:23:42:INFO:sockeye.training:__call__] E=42 B=168500	s/sec=1918.69 tok/sec=101894.71 u/sec=23.19	ppl=6.453696 
[2023-12-17:13:23:44:INFO:sockeye.training:__call__] E=42 B=168550	s/sec=2218.24 tok/sec=96412.43 u/sec=23.54	ppl=6.470175 
[2023-12-17:13:23:46:INFO:sockeye.training:__call__] E=42 B=168600	s/sec=1954.43 tok/sec=103759.76 u/sec=22.79	ppl=6.482918 
[2023-12-17:13:23:48:INFO:sockeye.training:__call__] E=42 B=168650	s/sec=2125.22 tok/sec=99727.62 u/sec=23.47	ppl=6.493516 
[2023-12-17:13:23:50:INFO:sockeye.training:__call__] E=42 B=168700	s/sec=2164.79 tok/sec=98599.59 u/sec=23.57	ppl=6.511535 
[2023-12-17:13:23:52:INFO:sockeye.training:__call__] E=42 B=168750	s/sec=1979.48 tok/sec=99581.74 u/sec=23.17	ppl=6.518231 
[2023-12-17:13:23:55:INFO:sockeye.training:__call__] E=42 B=168800	s/sec=1958.58 tok/sec=99440.35 u/sec=22.92	ppl=6.519952 
[2023-12-17:13:23:57:INFO:sockeye.training:__call__] E=42 B=168850	s/sec=2121.84 tok/sec=97634.53 u/sec=23.35	ppl=6.532736 
[2023-12-17:13:23:59:INFO:sockeye.training:__call__] E=42 B=168900	s/sec=2133.03 tok/sec=97211.71 u/sec=23.64	ppl=6.550252 
[2023-12-17:13:24:01:INFO:sockeye.training:__call__] E=42 B=168950	s/sec=2152.54 tok/sec=99829.47 u/sec=23.52	ppl=6.558385 
[2023-12-17:13:24:03:INFO:sockeye.training:__call__] E=42 B=169000	s/sec=2274.08 tok/sec=97714.33 u/sec=23.34	ppl=6.566204 
[2023-12-17:13:24:05:INFO:sockeye.training:__call__] E=42 B=169050	s/sec=2050.44 tok/sec=101194.08 u/sec=22.80	ppl=6.577105 
[2023-12-17:13:24:07:INFO:sockeye.training:__call__] E=42 B=169100	s/sec=2272.35 tok/sec=93754.83 u/sec=24.03	ppl=6.592352 
[2023-12-17:13:24:10:INFO:sockeye.training:__call__] E=42 B=169150	s/sec=1977.66 tok/sec=102564.29 u/sec=22.51	ppl=6.596592 
[2023-12-17:13:24:12:INFO:sockeye.training:__call__] E=42 B=169200	s/sec=1851.64 tok/sec=101745.56 u/sec=22.69	ppl=6.600904 
[2023-12-17:13:24:14:INFO:sockeye.training:__call__] E=42 B=169250	s/sec=2054.33 tok/sec=98847.13 u/sec=23.39	ppl=6.608858 
[2023-12-17:13:24:16:INFO:sockeye.training:__call__] E=42 B=169300	s/sec=2196.44 tok/sec=96049.66 u/sec=23.79	ppl=6.616402 
[2023-12-17:13:24:18:INFO:sockeye.training:__call__] E=42 B=169350	s/sec=2030.36 tok/sec=100989.76 u/sec=22.62	ppl=6.623652 
[2023-12-17:13:24:20:INFO:sockeye.training:__call__] E=42 B=169400	s/sec=1998.23 tok/sec=101184.12 u/sec=23.26	ppl=6.627983 
[2023-12-17:13:24:23:INFO:sockeye.training:__call__] E=42 B=169450	s/sec=2028.30 tok/sec=99580.46 u/sec=22.84	ppl=6.637586 
[2023-12-17:13:24:25:INFO:sockeye.training:__call__] E=42 B=169500	s/sec=2122.04 tok/sec=98738.58 u/sec=23.43	ppl=6.644237 
[2023-12-17:13:24:27:INFO:sockeye.training:__call__] E=42 B=169550	s/sec=2249.70 tok/sec=102054.73 u/sec=22.83	ppl=6.654331 
[2023-12-17:13:24:29:INFO:sockeye.training:__call__] E=42 B=169600	s/sec=2134.06 tok/sec=97328.23 u/sec=23.61	ppl=6.661881 
[2023-12-17:13:24:31:INFO:sockeye.training:__call__] E=42 B=169650	s/sec=2268.63 tok/sec=95359.72 u/sec=24.20	ppl=6.670004 
[2023-12-17:13:24:33:INFO:sockeye.training:__call__] E=42 B=169700	s/sec=2169.06 tok/sec=97887.39 u/sec=23.62	ppl=6.676400 
[2023-12-17:13:24:35:INFO:sockeye.training:__call__] E=42 B=169750	s/sec=2202.53 tok/sec=99867.70 u/sec=23.21	ppl=6.684046 
[2023-12-17:13:24:38:INFO:sockeye.training:__call__] E=42 B=169800	s/sec=2056.39 tok/sec=100060.07 u/sec=23.50	ppl=6.687210 
[2023-12-17:13:24:40:INFO:sockeye.training:__call__] E=42 B=169850	s/sec=2274.21 tok/sec=96861.70 u/sec=23.61	ppl=6.697126 
[2023-12-17:13:24:42:INFO:sockeye.training:__call__] E=42 B=169900	s/sec=1825.93 tok/sec=103065.47 u/sec=22.69	ppl=6.700372 
[2023-12-17:13:24:44:INFO:sockeye.training:__call__] E=42 B=169950	s/sec=1840.98 tok/sec=104084.53 u/sec=22.74	ppl=6.702272 
[2023-12-17:13:24:46:INFO:sockeye.training:__call__] E=42 B=170000	s/sec=1961.92 tok/sec=100657.84 u/sec=23.14	ppl=6.704605 
[2023-12-17:13:24:48:INFO:sockeye.training:__call__] E=42 B=170050	s/sec=2368.96 tok/sec=94850.38 u/sec=23.92	ppl=6.713332 
[2023-12-17:13:24:50:INFO:sockeye.training:__call__] E=42 B=170100	s/sec=2064.33 tok/sec=100538.12 u/sec=23.12	ppl=6.720062 
[2023-12-17:13:24:53:INFO:sockeye.training:__call__] E=42 B=170150	s/sec=2225.22 tok/sec=96405.81 u/sec=23.98	ppl=6.730427 
[2023-12-17:13:24:55:INFO:sockeye.training:__call__] E=42 B=170200	s/sec=2057.48 tok/sec=100446.47 u/sec=23.13	ppl=6.735803 
[2023-12-17:13:24:57:INFO:sockeye.training:__call__] E=42 B=170250	s/sec=2083.32 tok/sec=96818.73 u/sec=23.89	ppl=6.741760 
[2023-12-17:13:24:59:INFO:sockeye.training:__call__] E=42 B=170300	s/sec=2313.17 tok/sec=97767.45 u/sec=23.13	ppl=6.749636 
[2023-12-17:13:25:01:INFO:sockeye.training:__call__] E=42 B=170350	s/sec=2027.32 tok/sec=97819.26 u/sec=23.33	ppl=6.755566 
[2023-12-17:13:25:03:INFO:sockeye.training:__call__] E=42 B=170400	s/sec=2258.81 tok/sec=95502.24 u/sec=23.81	ppl=6.763564 
[2023-12-17:13:25:05:INFO:sockeye.training:__call__] E=42 B=170450	s/sec=1963.71 tok/sec=99968.03 u/sec=23.60	ppl=6.768162 
[2023-12-17:13:25:07:INFO:sockeye.training:__call__] E=42 B=170500	s/sec=2104.58 tok/sec=94748.50 u/sec=24.14	ppl=6.772804 
[2023-12-17:13:25:10:INFO:sockeye.training:__call__] E=42 B=170550	s/sec=2202.74 tok/sec=99055.65 u/sec=23.49	ppl=6.777495 
[2023-12-17:13:25:12:INFO:sockeye.training:__call__] E=42 B=170600	s/sec=2019.18 tok/sec=102757.08 u/sec=23.03	ppl=6.784047 
[2023-12-17:13:25:14:INFO:sockeye.training:__call__] E=42 B=170650	s/sec=2122.31 tok/sec=95541.98 u/sec=23.77	ppl=6.787746 
[2023-12-17:13:25:16:INFO:sockeye.training:__call__] E=42 B=170700	s/sec=1946.90 tok/sec=102546.31 u/sec=22.70	ppl=6.794873 
[2023-12-17:13:25:18:INFO:sockeye.training:__call__] E=42 B=170750	s/sec=2134.74 tok/sec=99177.52 u/sec=23.37	ppl=6.800429 
[2023-12-17:13:25:20:INFO:sockeye.training:__call__] E=42 B=170800	s/sec=1959.75 tok/sec=102252.86 u/sec=22.52	ppl=6.804887 
[2023-12-17:13:25:23:INFO:sockeye.training:__call__] E=42 B=170850	s/sec=1874.86 tok/sec=103501.05 u/sec=22.41	ppl=6.807784 
[2023-12-17:13:25:25:INFO:sockeye.training:__call__] E=42 B=170900	s/sec=1991.67 tok/sec=102729.59 u/sec=22.55	ppl=6.813122 
[2023-12-17:13:25:27:INFO:sockeye.training:__call__] E=42 B=170950	s/sec=2253.32 tok/sec=96731.34 u/sec=23.43	ppl=6.818501 
[2023-12-17:13:25:29:INFO:sockeye.training:__call__] E=42 B=171000	s/sec=2053.94 tok/sec=98014.12 u/sec=23.64	ppl=6.823223 
[2023-12-17:13:25:31:INFO:sockeye.training:__call__] E=42 B=171050	s/sec=2065.99 tok/sec=100699.55 u/sec=23.02	ppl=6.828761 
[2023-12-17:13:25:33:INFO:sockeye.training:__call__] E=42 B=171100	s/sec=1716.31 tok/sec=107212.47 u/sec=22.35	ppl=6.829751 
[2023-12-17:13:25:36:INFO:sockeye.training:__call__] E=42 B=171150	s/sec=1849.60 tok/sec=102922.47 u/sec=23.26	ppl=6.833155 
[2023-12-17:13:25:38:INFO:sockeye.training:__call__] E=42 B=171200	s/sec=1909.70 tok/sec=102867.38 u/sec=22.91	ppl=6.835193 
[2023-12-17:13:25:40:INFO:sockeye.training:__call__] E=42 B=171250	s/sec=2030.42 tok/sec=100416.15 u/sec=23.33	ppl=6.839273 
[2023-12-17:13:25:42:INFO:sockeye.training:__call__] E=42 B=171300	s/sec=2021.82 tok/sec=99374.83 u/sec=23.31	ppl=6.844536 
[2023-12-17:13:25:44:INFO:sockeye.training:__call__] E=42 B=171350	s/sec=2224.61 tok/sec=97174.99 u/sec=23.53	ppl=6.849273 
[2023-12-17:13:25:46:INFO:sockeye.training:__call__] E=42 B=171400	s/sec=1942.08 tok/sec=100585.59 u/sec=23.30	ppl=6.854977 
[2023-12-17:13:25:48:INFO:sockeye.training:__call__] E=42 B=171450	s/sec=2207.85 tok/sec=97450.01 u/sec=23.79	ppl=6.857870 
[2023-12-17:13:25:51:INFO:sockeye.training:__call__] E=42 B=171500	s/sec=2062.24 tok/sec=100136.71 u/sec=23.02	ppl=6.863324 
[2023-12-17:13:25:53:INFO:sockeye.training:__call__] E=42 B=171550	s/sec=1952.99 tok/sec=103551.35 u/sec=22.27	ppl=6.867955 
[2023-12-17:13:25:55:INFO:sockeye.training:__call__] E=42 B=171600	s/sec=1954.46 tok/sec=100693.80 u/sec=23.27	ppl=6.873399 
[2023-12-17:13:25:57:INFO:sockeye.training:__call__] E=42 B=171650	s/sec=2006.37 tok/sec=101068.21 u/sec=22.97	ppl=6.878260 
[2023-12-17:13:25:59:INFO:sockeye.training:__call__] E=42 B=171700	s/sec=2192.57 tok/sec=100066.90 u/sec=23.11	ppl=6.882217 
[2023-12-17:13:26:01:INFO:sockeye.training:__call__] E=42 B=171750	s/sec=2298.69 tok/sec=96453.84 u/sec=23.79	ppl=6.887677 
[2023-12-17:13:26:04:INFO:sockeye.training:__call__] E=43 B=171800	s/sec=2312.47 tok/sec=89235.76 u/sec=22.76	ppl=6.889306 
[2023-12-17:13:26:06:INFO:sockeye.training:__call__] E=43 B=171850	s/sec=1939.58 tok/sec=98750.42 u/sec=23.36	ppl=6.879438 
[2023-12-17:13:26:08:INFO:sockeye.training:__call__] E=43 B=171900	s/sec=2287.26 tok/sec=95040.06 u/sec=23.91	ppl=6.871347 
[2023-12-17:13:26:10:INFO:sockeye.training:__call__] E=43 B=171950	s/sec=2241.51 tok/sec=95951.73 u/sec=23.74	ppl=6.863835 
[2023-12-17:13:26:12:INFO:sockeye.training:__call__] E=43 B=172000	s/sec=2061.23 tok/sec=100992.97 u/sec=23.25	ppl=6.855880 
[2023-12-17:13:26:12:INFO:sockeye.training:_create_checkpoint] Checkpoint [43]	Updates=172000 Epoch=43 Samples=15329344 Time-cost=172.005 Updates/sec=23.255
[2023-12-17:13:26:12:INFO:sockeye.training:_create_checkpoint] Checkpoint [43]	Train-ppl=6.855880
[2023-12-17:13:26:15:INFO:sockeye.training:_evaluate] Checkpoint [43]	Validation-ppl=30.971902	Validation-bleu=0.161286	Validation-chrf=30.904963	Validation-rouge1=0.159615	Validation-rouge2=0.075155	Validation-rougel=0.152279	Validation-length-ratio-mse=1.282078	Validation-ter=96.491952	Validation-avg-sec-per-sent=0.003414	Validation-decode-walltime=1.706895
[2023-12-17:13:26:15:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 19 checkpoints, best so far: 32.780032
[2023-12-17:13:26:15:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints
[2023-12-17:13:26:15:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [43]	Learning-rate=0.000131
[2023-12-17:13:26:16:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:26:16:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:26:16:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00043"
[2023-12-17:13:26:18:INFO:sockeye.training:__call__] E=43 B=172050	s/sec=727.77 tok/sec=36448.56 u/sec=8.52	ppl=6.228267 
[2023-12-17:13:26:20:INFO:sockeye.training:__call__] E=43 B=172100	s/sec=2226.37 tok/sec=97688.75 u/sec=23.58	ppl=6.280219 
[2023-12-17:13:26:22:INFO:sockeye.training:__call__] E=43 B=172150	s/sec=1994.81 tok/sec=100758.26 u/sec=22.88	ppl=6.271898 
[2023-12-17:13:26:24:INFO:sockeye.training:__call__] E=43 B=172200	s/sec=2187.29 tok/sec=98461.51 u/sec=23.29	ppl=6.298535 
[2023-12-17:13:26:27:INFO:sockeye.training:__call__] E=43 B=172250	s/sec=2142.19 tok/sec=97854.60 u/sec=23.82	ppl=6.326034 
[2023-12-17:13:26:29:INFO:sockeye.training:__call__] E=43 B=172300	s/sec=2310.43 tok/sec=96735.89 u/sec=23.60	ppl=6.343915 
[2023-12-17:13:26:31:INFO:sockeye.training:__call__] E=43 B=172350	s/sec=1981.45 tok/sec=99057.49 u/sec=23.37	ppl=6.331440 
[2023-12-17:13:26:33:INFO:sockeye.training:__call__] E=43 B=172400	s/sec=2119.31 tok/sec=98252.22 u/sec=23.40	ppl=6.341256 
[2023-12-17:13:26:35:INFO:sockeye.training:__call__] E=43 B=172450	s/sec=2150.87 tok/sec=96990.05 u/sec=23.26	ppl=6.348259 
[2023-12-17:13:26:37:INFO:sockeye.training:__call__] E=43 B=172500	s/sec=2103.38 tok/sec=96710.79 u/sec=23.23	ppl=6.359650 
[2023-12-17:13:26:39:INFO:sockeye.training:__call__] E=43 B=172550	s/sec=2115.00 tok/sec=101047.51 u/sec=23.07	ppl=6.366293 
[2023-12-17:13:26:42:INFO:sockeye.training:__call__] E=43 B=172600	s/sec=1772.48 tok/sec=102136.15 u/sec=22.94	ppl=6.366681 
[2023-12-17:13:26:44:INFO:sockeye.training:__call__] E=43 B=172650	s/sec=1994.13 tok/sec=100483.34 u/sec=23.12	ppl=6.379698 
[2023-12-17:13:26:46:INFO:sockeye.training:__call__] E=43 B=172700	s/sec=1982.18 tok/sec=99860.81 u/sec=23.03	ppl=6.382433 
[2023-12-17:13:26:48:INFO:sockeye.training:__call__] E=43 B=172750	s/sec=2203.82 tok/sec=98088.36 u/sec=23.03	ppl=6.396126 
[2023-12-17:13:26:50:INFO:sockeye.training:__call__] E=43 B=172800	s/sec=2174.87 tok/sec=96319.04 u/sec=23.81	ppl=6.400281 
[2023-12-17:13:26:52:INFO:sockeye.training:__call__] E=43 B=172850	s/sec=2044.71 tok/sec=100461.74 u/sec=23.19	ppl=6.406406 
[2023-12-17:13:26:55:INFO:sockeye.training:__call__] E=43 B=172900	s/sec=1985.59 tok/sec=101709.87 u/sec=23.15	ppl=6.412292 
[2023-12-17:13:26:57:INFO:sockeye.training:__call__] E=43 B=172950	s/sec=2042.05 tok/sec=102233.24 u/sec=22.71	ppl=6.418584 
[2023-12-17:13:26:59:INFO:sockeye.training:__call__] E=43 B=173000	s/sec=2147.15 tok/sec=93954.87 u/sec=23.88	ppl=6.426932 
[2023-12-17:13:27:01:INFO:sockeye.training:__call__] E=43 B=173050	s/sec=2296.81 tok/sec=94970.27 u/sec=23.73	ppl=6.439954 
[2023-12-17:13:27:03:INFO:sockeye.training:__call__] E=43 B=173100	s/sec=2119.95 tok/sec=95345.16 u/sec=23.41	ppl=6.450039 
[2023-12-17:13:27:05:INFO:sockeye.training:__call__] E=43 B=173150	s/sec=1813.65 tok/sec=99910.21 u/sec=22.67	ppl=6.454536 
[2023-12-17:13:27:07:INFO:sockeye.training:__call__] E=43 B=173200	s/sec=1946.76 tok/sec=98764.84 u/sec=23.22	ppl=6.456337 
[2023-12-17:13:27:10:INFO:sockeye.training:__call__] E=43 B=173250	s/sec=1966.67 tok/sec=99437.41 u/sec=23.15	ppl=6.463498 
[2023-12-17:13:27:12:INFO:sockeye.training:__call__] E=43 B=173300	s/sec=2049.54 tok/sec=100584.08 u/sec=23.29	ppl=6.467665 
[2023-12-17:13:27:14:INFO:sockeye.training:__call__] E=43 B=173350	s/sec=2178.20 tok/sec=98302.93 u/sec=23.04	ppl=6.477452 
[2023-12-17:13:27:16:INFO:sockeye.training:__call__] E=43 B=173400	s/sec=2023.03 tok/sec=99832.72 u/sec=23.41	ppl=6.479887 
[2023-12-17:13:27:18:INFO:sockeye.training:__call__] E=43 B=173450	s/sec=1981.94 tok/sec=103569.86 u/sec=22.73	ppl=6.483331 
[2023-12-17:13:27:20:INFO:sockeye.training:__call__] E=43 B=173500	s/sec=1983.12 tok/sec=100034.94 u/sec=23.08	ppl=6.490029 
[2023-12-17:13:27:23:INFO:sockeye.training:__call__] E=43 B=173550	s/sec=2093.78 tok/sec=99019.06 u/sec=23.16	ppl=6.495565 
[2023-12-17:13:27:25:INFO:sockeye.training:__call__] E=43 B=173600	s/sec=2000.76 tok/sec=97600.90 u/sec=23.51	ppl=6.501228 
[2023-12-17:13:27:27:INFO:sockeye.training:__call__] E=43 B=173650	s/sec=2033.99 tok/sec=98862.51 u/sec=23.45	ppl=6.507297 
[2023-12-17:13:27:29:INFO:sockeye.training:__call__] E=43 B=173700	s/sec=1907.02 tok/sec=104825.60 u/sec=22.28	ppl=6.509706 
[2023-12-17:13:27:31:INFO:sockeye.training:__call__] E=43 B=173750	s/sec=2067.90 tok/sec=100356.31 u/sec=23.16	ppl=6.516763 
[2023-12-17:13:27:33:INFO:sockeye.training:__call__] E=43 B=173800	s/sec=2002.66 tok/sec=98305.94 u/sec=22.39	ppl=6.521311 
[2023-12-17:13:27:36:INFO:sockeye.training:__call__] E=43 B=173850	s/sec=2410.92 tok/sec=92202.37 u/sec=24.15	ppl=6.533350 
[2023-12-17:13:27:38:INFO:sockeye.training:__call__] E=43 B=173900	s/sec=1896.18 tok/sec=100590.91 u/sec=23.37	ppl=6.534556 
[2023-12-17:13:27:40:INFO:sockeye.training:__call__] E=43 B=173950	s/sec=2022.81 tok/sec=101608.29 u/sec=23.20	ppl=6.542008 
[2023-12-17:13:27:42:INFO:sockeye.training:__call__] E=43 B=174000	s/sec=1965.51 tok/sec=100307.87 u/sec=23.18	ppl=6.546760 
[2023-12-17:13:27:44:INFO:sockeye.training:__call__] E=43 B=174050	s/sec=2087.48 tok/sec=102153.52 u/sec=23.13	ppl=6.551892 
[2023-12-17:13:27:46:INFO:sockeye.training:__call__] E=43 B=174100	s/sec=1818.13 tok/sec=101815.46 u/sec=22.91	ppl=6.556052 
[2023-12-17:13:27:49:INFO:sockeye.training:__call__] E=43 B=174150	s/sec=2212.80 tok/sec=103411.78 u/sec=22.34	ppl=6.561441 
[2023-12-17:13:27:51:INFO:sockeye.training:__call__] E=43 B=174200	s/sec=1910.36 tok/sec=100961.39 u/sec=23.05	ppl=6.567744 
[2023-12-17:13:27:53:INFO:sockeye.training:__call__] E=43 B=174250	s/sec=2167.54 tok/sec=96854.84 u/sec=23.89	ppl=6.572402 
[2023-12-17:13:27:55:INFO:sockeye.training:__call__] E=43 B=174300	s/sec=1900.60 tok/sec=98357.02 u/sec=22.46	ppl=6.576038 
[2023-12-17:13:27:57:INFO:sockeye.training:__call__] E=43 B=174350	s/sec=1982.37 tok/sec=102405.27 u/sec=23.03	ppl=6.580490 
[2023-12-17:13:27:59:INFO:sockeye.training:__call__] E=43 B=174400	s/sec=2158.25 tok/sec=96726.24 u/sec=23.75	ppl=6.590258 
[2023-12-17:13:28:02:INFO:sockeye.training:__call__] E=43 B=174450	s/sec=2017.21 tok/sec=102604.23 u/sec=22.80	ppl=6.597032 
[2023-12-17:13:28:04:INFO:sockeye.training:__call__] E=43 B=174500	s/sec=2035.85 tok/sec=100705.71 u/sec=22.64	ppl=6.601711 
[2023-12-17:13:28:06:INFO:sockeye.training:__call__] E=43 B=174550	s/sec=2294.65 tok/sec=97581.37 u/sec=23.71	ppl=6.607517 
[2023-12-17:13:28:08:INFO:sockeye.training:__call__] E=43 B=174600	s/sec=2015.30 tok/sec=102089.48 u/sec=22.37	ppl=6.611036 
[2023-12-17:13:28:10:INFO:sockeye.training:__call__] E=43 B=174650	s/sec=2236.82 tok/sec=98074.02 u/sec=23.50	ppl=6.617879 
[2023-12-17:13:28:12:INFO:sockeye.training:__call__] E=43 B=174700	s/sec=1929.72 tok/sec=100270.03 u/sec=23.51	ppl=6.620801 
[2023-12-17:13:28:14:INFO:sockeye.training:__call__] E=43 B=174750	s/sec=2273.13 tok/sec=96090.94 u/sec=23.33	ppl=6.624834 
[2023-12-17:13:28:17:INFO:sockeye.training:__call__] E=43 B=174800	s/sec=1871.75 tok/sec=103167.08 u/sec=23.03	ppl=6.628269 
[2023-12-17:13:28:19:INFO:sockeye.training:__call__] E=43 B=174850	s/sec=2003.11 tok/sec=99668.52 u/sec=22.72	ppl=6.632921 
[2023-12-17:13:28:21:INFO:sockeye.training:__call__] E=43 B=174900	s/sec=2202.50 tok/sec=95795.52 u/sec=23.86	ppl=6.637675 
[2023-12-17:13:28:23:INFO:sockeye.training:__call__] E=43 B=174950	s/sec=1999.84 tok/sec=100508.67 u/sec=23.06	ppl=6.642272 
[2023-12-17:13:28:25:INFO:sockeye.training:__call__] E=43 B=175000	s/sec=2020.56 tok/sec=102108.42 u/sec=23.13	ppl=6.647497 
[2023-12-17:13:28:27:INFO:sockeye.training:__call__] E=43 B=175050	s/sec=1749.04 tok/sec=101797.16 u/sec=22.97	ppl=6.650322 
[2023-12-17:13:28:30:INFO:sockeye.training:__call__] E=43 B=175100	s/sec=2085.96 tok/sec=98382.30 u/sec=23.49	ppl=6.655827 
[2023-12-17:13:28:32:INFO:sockeye.training:__call__] E=43 B=175150	s/sec=1921.60 tok/sec=102051.12 u/sec=22.62	ppl=6.660390 
[2023-12-17:13:28:34:INFO:sockeye.training:__call__] E=43 B=175200	s/sec=2158.54 tok/sec=96402.17 u/sec=23.96	ppl=6.667240 
[2023-12-17:13:28:36:INFO:sockeye.training:__call__] E=43 B=175250	s/sec=2283.39 tok/sec=97838.07 u/sec=23.63	ppl=6.673964 
[2023-12-17:13:28:38:INFO:sockeye.training:__call__] E=43 B=175300	s/sec=2110.26 tok/sec=103976.17 u/sec=22.55	ppl=6.678846 
[2023-12-17:13:28:40:INFO:sockeye.training:__call__] E=43 B=175350	s/sec=2120.02 tok/sec=97626.65 u/sec=23.58	ppl=6.684197 
[2023-12-17:13:28:42:INFO:sockeye.training:__call__] E=43 B=175400	s/sec=2245.57 tok/sec=97887.45 u/sec=23.51	ppl=6.690673 
[2023-12-17:13:28:45:INFO:sockeye.training:__call__] E=43 B=175450	s/sec=2181.77 tok/sec=98281.59 u/sec=23.63	ppl=6.696090 
[2023-12-17:13:28:47:INFO:sockeye.training:__call__] E=43 B=175500	s/sec=2164.61 tok/sec=101644.90 u/sec=22.89	ppl=6.700966 
[2023-12-17:13:28:49:INFO:sockeye.training:__call__] E=43 B=175550	s/sec=2324.69 tok/sec=95132.44 u/sec=23.94	ppl=6.707584 
[2023-12-17:13:28:51:INFO:sockeye.training:__call__] E=43 B=175600	s/sec=1823.61 tok/sec=103388.26 u/sec=22.48	ppl=6.710000 
[2023-12-17:13:28:53:INFO:sockeye.training:__call__] E=43 B=175650	s/sec=1943.47 tok/sec=99814.13 u/sec=23.18	ppl=6.714193 
[2023-12-17:13:28:55:INFO:sockeye.training:__call__] E=43 B=175700	s/sec=2068.99 tok/sec=99221.32 u/sec=23.47	ppl=6.718784 
[2023-12-17:13:28:58:INFO:sockeye.training:__call__] E=43 B=175750	s/sec=2185.17 tok/sec=98234.57 u/sec=23.63	ppl=6.725010 
[2023-12-17:13:29:00:INFO:sockeye.training:__call__] E=44 B=175800	s/sec=2068.07 tok/sec=97370.31 u/sec=21.80	ppl=6.723902 
[2023-12-17:13:29:02:INFO:sockeye.training:__call__] E=44 B=175850	s/sec=2020.30 tok/sec=96854.37 u/sec=23.47	ppl=6.716118 
[2023-12-17:13:29:04:INFO:sockeye.training:__call__] E=44 B=175900	s/sec=2022.73 tok/sec=100884.79 u/sec=23.15	ppl=6.707155 
[2023-12-17:13:29:06:INFO:sockeye.training:__call__] E=44 B=175950	s/sec=2125.03 tok/sec=95802.81 u/sec=23.97	ppl=6.699314 
[2023-12-17:13:29:08:INFO:sockeye.training:__call__] E=44 B=176000	s/sec=2096.48 tok/sec=97939.13 u/sec=23.11	ppl=6.691794 
[2023-12-17:13:29:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [44]	Updates=176000 Epoch=44 Samples=15685328 Time-cost=172.473 Updates/sec=23.192
[2023-12-17:13:29:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [44]	Train-ppl=6.691794
[2023-12-17:13:29:11:INFO:sockeye.training:_evaluate] Checkpoint [44]	Validation-ppl=31.778880	Validation-bleu=0.172071	Validation-chrf=31.356439	Validation-rouge1=0.163102	Validation-rouge2=0.080321	Validation-rougel=0.156992	Validation-length-ratio-mse=1.278343	Validation-ter=95.418902	Validation-avg-sec-per-sent=0.003258	Validation-decode-walltime=1.629090
[2023-12-17:13:29:11:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 20 checkpoints, best so far: 32.780032
[2023-12-17:13:29:11:INFO:sockeye.training:_determine_convergence] Maximum number of not improved checkpoints reached: improvement 0.000000 <= 0.000000 over 20 checkpoints
[2023-12-17:13:29:11:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [44]	Learning-rate=0.000131
[2023-12-17:13:29:12:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl"
[2023-12-17:13:29:12:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl'
[2023-12-17:13:29:12:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00044"
[2023-12-17:13:29:12:INFO:sockeye.training:fit] Training finished. Best checkpoint: 24. Best validation chrf: 32.780032