2024-08-07 19:01:43,343 - INFO: Calling run..
2024-08-07 19:01:43,344 - INFO: Environment configuration: ConfigNLPCausalClassificationEnvironment(gpus=['0'], mixed_precision=False, compile_model=False, use_deepspeed=False, deepspeed_reduce_bucket_size=10000000.0, deepspeed_stage3_prefetch_bucket_size=10000000.0, deepspeed_stage3_param_persistence_threshold=10000000.0, deepspeed_offload_optimizer=False, deepspeed_stage3_max_live_parameters=10000000.0, deepspeed_stage3_max_reuse_distance=10000000.0, find_unused_parameters=False, trust_remote_code=False, huggingface_branch='main', number_of_workers=8, seed=-1, _seed=0, _distributed=False, _distributed_inference=True, _local_rank=0, _world_size=1, _curr_step=0, _curr_val_step=0, _rank=0, _device='cuda', _cpu_comm=None, _model_card_template='text_causal_classification_model_card_template.md', _summary_card_template='text_causal_classification_experiment_summary_card_template.md')
2024-08-07 19:01:43,344 - INFO: cfg.environment._distributed set to False
2024-08-07 19:01:43,345 - INFO: Problem Type: text_causal_classification_modeling
2024-08-07 19:01:43,345 - INFO: Global random seed: 288462
2024-08-07 19:01:43,345 - INFO: Preparing the data...
2024-08-07 19:01:43,346 - INFO: Setting up automatic validation split...
2024-08-07 19:01:43,381 - INFO: The dataframe has following columns: Index(['Description', 'category', 'sub_category', 'label'], dtype='object')
2024-08-07 19:01:43,387 - INFO: Preparing train and validation data, dataset config to be used: ConfigNLPCausalClassificationDataset(dataset_class=<class 'llm_studio.src.datasets.text_causal_classification_ds.CustomDataset'>, personalize=False, chatbot_name='OI_AI', chatbot_author='openinnovation.ai', train_dataframe='/app/train_df.csv', validation_strategy='automatic', validation_dataframe='', validation_size=0.2, data_sample=1.0, data_sample_choice=('Train', 'Validation'), system_column='None', prompt_column=('Description',), answer_column='category', parent_id_column='None', text_system_start='', text_prompt_start='', text_answer_separator='', limit_chained_samples=False, add_eos_token_to_system=False, add_eos_token_to_prompt=False, add_eos_token_to_answer=False, mask_prompt_labels=True, _allowed_file_extensions=('csv', 'pq', 'parquet'), num_classes=10)
2024-08-07 19:01:43,387 - INFO: Loading train dataset...
2024-08-07 19:01:43,387 - INFO: Columns found: Index(['Description', 'category', 'sub_category', 'label'], dtype='object')
2024-08-07 19:01:44,557 - INFO: Loading validation dataset...
2024-08-07 19:01:44,706 - INFO: Number of observations in train dataset: 9600
2024-08-07 19:01:44,706 - INFO: Number of observations in validation dataset: 2400
2024-08-07 19:01:44,965 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id.
2024-08-07 19:01:44,973 - INFO: Using float16 for backbone
2024-08-07 19:01:44,973 - INFO: Loading tiiuae/falcon-7b. This may take a while.
2024-08-07 19:03:58,808 - INFO: Loaded tiiuae/falcon-7b.
2024-08-07 19:03:58,810 - WARNING: PAD token id not matching between generation config and tokenizer. Overwriting with tokenizer id.
2024-08-07 19:03:58,810 - INFO: Lora module names: ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h']
2024-08-07 19:03:58,943 - INFO: Enough space available for saving model weights.Required space: 13632.81MB, Available space: 368261.23MB.
2024-08-07 19:03:58,947 - INFO: Optimizer AdamW has been provided with parameters {'weight_decay': 0.0, 'eps': 1e-08, 'betas': (0.8999999762, 0.9990000129), 'lr': 0.0001}
2024-08-07 19:03:59,367 - INFO: started process: 0, can_track: True, tracking_mode: TrackingMode.AFTER_EPOCH
2024-08-07 19:03:59,368 - INFO: Training Epoch: 1 / 10
2024-08-07 19:03:59,369 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 19:03:59,533 - INFO: Evaluation step: 300
2024-08-07 19:04:21,347 - INFO: train loss: 42.78:   1%|1         | 3/300 [00:21<36:15,  7.33s/it]
2024-08-07 19:04:26,648 - INFO: train loss: 66.63:   2%|2         | 6/300 [00:27<19:52,  4.06s/it]
2024-08-07 19:04:31,837 - INFO: train loss: 88.90:   3%|3         | 9/300 [00:32<14:31,  2.99s/it]
2024-08-07 19:04:36,782 - INFO: train loss: 92.28:   4%|4         | 12/300 [00:37<11:49,  2.46s/it]
2024-08-07 19:04:41,794 - INFO: train loss: 76.47:   5%|5         | 15/300 [00:42<10:20,  2.18s/it]
2024-08-07 19:04:47,149 - INFO: train loss: 43.56:   6%|6         | 18/300 [00:47<09:36,  2.04s/it]
2024-08-07 19:04:51,908 - INFO: train loss: 18.78:   7%|7         | 21/300 [00:52<08:48,  1.89s/it]
2024-08-07 19:04:56,631 - INFO: train loss: 8.20:   8%|8         | 24/300 [00:57<08:14,  1.79s/it]
2024-08-07 19:05:01,294 - INFO: train loss: 5.85:   9%|9         | 27/300 [01:01<07:48,  1.72s/it]
2024-08-07 19:05:06,377 - INFO: train loss: 3.56:  10%|#         | 30/300 [01:07<07:41,  1.71s/it]
2024-08-07 19:05:11,288 - INFO: train loss: 3.24:  11%|#1        | 33/300 [01:11<07:30,  1.69s/it]
2024-08-07 19:05:16,310 - INFO: train loss: 3.20:  12%|#2        | 36/300 [01:16<07:24,  1.68s/it]
2024-08-07 19:05:21,802 - INFO: train loss: 2.91:  13%|#3        | 39/300 [01:22<07:31,  1.73s/it]
2024-08-07 19:05:26,717 - INFO: train loss: 2.94:  14%|#4        | 42/300 [01:27<07:18,  1.70s/it]
2024-08-07 19:05:31,630 - INFO: train loss: 2.52:  15%|#5        | 45/300 [01:32<07:08,  1.68s/it]
2024-08-07 19:05:36,474 - INFO: train loss: 2.30:  16%|#6        | 48/300 [01:37<06:58,  1.66s/it]
2024-08-07 19:05:41,478 - INFO: train loss: 1.99:  17%|#7        | 51/300 [01:42<06:54,  1.66s/it]
2024-08-07 19:05:46,191 - INFO: train loss: 1.84:  18%|#8        | 54/300 [01:46<06:42,  1.64s/it]
2024-08-07 19:05:51,465 - INFO: train loss: 1.82:  19%|#9        | 57/300 [01:52<06:46,  1.67s/it]
2024-08-07 19:05:56,388 - INFO: train loss: 2.09:  20%|##        | 60/300 [01:57<06:39,  1.66s/it]
2024-08-07 19:06:01,888 - INFO: train loss: 2.33:  21%|##1       | 63/300 [02:02<06:46,  1.71s/it]
2024-08-07 19:06:06,594 - INFO: train loss: 2.30:  22%|##2       | 66/300 [02:07<06:30,  1.67s/it]
2024-08-07 19:06:11,394 - INFO: train loss: 2.12:  23%|##3       | 69/300 [02:12<06:20,  1.65s/it]
2024-08-07 19:06:16,189 - INFO: train loss: 1.77:  24%|##4       | 72/300 [02:16<06:12,  1.63s/it]
2024-08-07 19:06:20,994 - INFO: train loss: 1.69:  25%|##5       | 75/300 [02:21<06:05,  1.62s/it]
2024-08-07 19:06:25,706 - INFO: train loss: 1.54:  26%|##6       | 78/300 [02:26<05:57,  1.61s/it]
2024-08-07 19:06:30,764 - INFO: train loss: 1.55:  27%|##7       | 81/300 [02:31<05:57,  1.63s/it]
2024-08-07 19:06:35,446 - INFO: train loss: 1.55:  28%|##8       | 84/300 [02:36<05:47,  1.61s/it]
2024-08-07 19:06:40,842 - INFO: train loss: 1.58:  29%|##9       | 87/300 [02:41<05:55,  1.67s/it]
2024-08-07 19:06:45,584 - INFO: train loss: 1.60:  30%|###       | 90/300 [02:46<05:44,  1.64s/it]
2024-08-07 19:06:50,936 - INFO: train loss: 1.95:  31%|###1      | 93/300 [02:51<05:48,  1.68s/it]
2024-08-07 19:06:55,947 - INFO: train loss: 1.98:  32%|###2      | 96/300 [02:56<05:42,  1.68s/it]
2024-08-07 19:07:01,137 - INFO: train loss: 2.24:  33%|###3      | 99/300 [03:01<05:40,  1.69s/it]
2024-08-07 19:07:05,556 - INFO: train loss: 1.71:  34%|###4      | 102/300 [03:06<05:22,  1.63s/it]
2024-08-07 19:07:10,290 - INFO: train loss: 1.80:  35%|###5      | 105/300 [03:10<05:14,  1.61s/it]
2024-08-07 19:07:15,450 - INFO: train loss: 1.43:  36%|###6      | 108/300 [03:16<05:15,  1.65s/it]
2024-08-07 19:07:20,182 - INFO: train loss: 1.65:  37%|###7      | 111/300 [03:20<05:07,  1.62s/it]
2024-08-07 19:07:25,095 - INFO: train loss: 1.56:  38%|###8      | 114/300 [03:25<05:02,  1.63s/it]
2024-08-07 19:07:30,293 - INFO: train loss: 1.41:  39%|###9      | 117/300 [03:30<05:03,  1.66s/it]
2024-08-07 19:07:35,139 - INFO: train loss: 1.21:  40%|####      | 120/300 [03:35<04:56,  1.65s/it]
2024-08-07 19:07:40,126 - INFO: train loss: 1.03:  41%|####1     | 123/300 [03:40<04:52,  1.65s/it]
2024-08-07 19:07:45,181 - INFO: train loss: 1.09:  42%|####2     | 126/300 [03:45<04:49,  1.66s/it]
2024-08-07 19:07:50,224 - INFO: train loss: 1.01:  43%|####3     | 129/300 [03:50<04:45,  1.67s/it]
2024-08-07 19:07:55,119 - INFO: train loss: 1.06:  44%|####4     | 132/300 [03:55<04:38,  1.66s/it]
2024-08-07 19:08:00,282 - INFO: train loss: 0.90:  45%|####5     | 135/300 [04:00<04:36,  1.68s/it]
2024-08-07 19:08:05,376 - INFO: train loss: 0.78:  46%|####6     | 138/300 [04:06<04:32,  1.68s/it]
2024-08-07 19:08:10,200 - INFO: train loss: 0.68:  47%|####6     | 141/300 [04:10<04:23,  1.66s/it]
2024-08-07 19:08:15,038 - INFO: train loss: 0.77:  48%|####8     | 144/300 [04:15<04:16,  1.65s/it]
2024-08-07 19:08:20,027 - INFO: train loss: 1.08:  49%|####9     | 147/300 [04:20<04:12,  1.65s/it]
2024-08-07 19:08:25,059 - INFO: train loss: 1.07:  50%|#####     | 150/300 [04:25<04:08,  1.66s/it]
2024-08-07 19:08:30,150 - INFO: train loss: 1.16:  51%|#####1    | 153/300 [04:30<04:05,  1.67s/it]
2024-08-07 19:08:35,214 - INFO: train loss: 0.99:  52%|#####2    | 156/300 [04:35<04:01,  1.68s/it]
2024-08-07 19:08:40,371 - INFO: train loss: 0.99:  53%|#####3    | 159/300 [04:41<03:58,  1.69s/it]
2024-08-07 19:08:45,405 - INFO: train loss: 1.09:  54%|#####4    | 162/300 [04:46<03:52,  1.69s/it]
2024-08-07 19:08:50,067 - INFO: train loss: 0.97:  55%|#####5    | 165/300 [04:50<03:42,  1.65s/it]
2024-08-07 19:08:55,289 - INFO: train loss: 1.05:  56%|#####6    | 168/300 [04:55<03:41,  1.67s/it]
2024-08-07 19:09:00,586 - INFO: train loss: 0.80:  57%|#####6    | 171/300 [05:01<03:39,  1.70s/it]
2024-08-07 19:09:05,547 - INFO: train loss: 0.78:  58%|#####8    | 174/300 [05:06<03:32,  1.69s/it]
2024-08-07 19:09:10,638 - INFO: train loss: 0.64:  59%|#####8    | 177/300 [05:11<03:27,  1.69s/it]
2024-08-07 19:09:15,498 - INFO: train loss: 0.63:  60%|######    | 180/300 [05:16<03:20,  1.67s/it]
2024-08-07 19:09:20,402 - INFO: train loss: 0.63:  61%|######1   | 183/300 [05:21<03:14,  1.66s/it]
2024-08-07 19:09:26,507 - INFO: train loss: 0.70:  62%|######2   | 186/300 [05:27<03:21,  1.77s/it]
2024-08-07 19:09:31,511 - INFO: train loss: 0.79:  63%|######3   | 189/300 [05:32<03:13,  1.74s/it]
2024-08-07 19:09:36,109 - INFO: train loss: 0.87:  64%|######4   | 192/300 [05:36<03:01,  1.68s/it]
2024-08-07 19:09:41,658 - INFO: train loss: 0.68:  65%|######5   | 195/300 [05:42<03:01,  1.73s/it]
2024-08-07 19:09:46,689 - INFO: train loss: 0.87:  66%|######6   | 198/300 [05:47<02:54,  1.71s/it]
2024-08-07 19:09:51,832 - INFO: train loss: 0.92:  67%|######7   | 201/300 [05:52<02:49,  1.71s/it]
2024-08-07 19:09:56,511 - INFO: train loss: 1.01:  68%|######8   | 204/300 [05:57<02:40,  1.67s/it]
2024-08-07 19:10:01,305 - INFO: train loss: 1.04:  69%|######9   | 207/300 [06:01<02:33,  1.65s/it]
2024-08-07 19:10:06,511 - INFO: train loss: 0.90:  70%|#######   | 210/300 [06:07<02:30,  1.67s/it]
2024-08-07 19:10:11,465 - INFO: train loss: 0.80:  71%|#######1  | 213/300 [06:12<02:25,  1.67s/it]
2024-08-07 19:10:16,589 - INFO: train loss: 0.80:  72%|#######2  | 216/300 [06:17<02:21,  1.68s/it]
2024-08-07 19:10:21,452 - INFO: train loss: 0.82:  73%|#######3  | 219/300 [06:22<02:14,  1.66s/it]
2024-08-07 19:10:26,417 - INFO: train loss: 0.85:  74%|#######4  | 222/300 [06:27<02:09,  1.66s/it]
2024-08-07 19:10:31,503 - INFO: train loss: 0.87:  75%|#######5  | 225/300 [06:32<02:05,  1.67s/it]
2024-08-07 19:10:36,396 - INFO: train loss: 0.70:  76%|#######6  | 228/300 [06:37<01:59,  1.66s/it]
2024-08-07 19:10:41,514 - INFO: train loss: 0.65:  77%|#######7  | 231/300 [06:42<01:55,  1.67s/it]
2024-08-07 19:10:46,341 - INFO: train loss: 0.52:  78%|#######8  | 234/300 [06:46<01:49,  1.65s/it]
2024-08-07 19:10:51,510 - INFO: train loss: 0.56:  79%|#######9  | 237/300 [06:52<01:45,  1.67s/it]
2024-08-07 19:10:56,430 - INFO: train loss: 0.69:  80%|########  | 240/300 [06:57<01:39,  1.66s/it]
2024-08-07 19:11:01,795 - INFO: train loss: 0.88:  81%|########1 | 243/300 [07:02<01:36,  1.70s/it]
2024-08-07 19:11:07,429 - INFO: train loss: 0.84:  82%|########2 | 246/300 [07:08<01:34,  1.75s/it]
2024-08-07 19:11:12,323 - INFO: train loss: 0.74:  83%|########2 | 249/300 [07:12<01:27,  1.72s/it]
2024-08-07 19:11:17,576 - INFO: train loss: 0.60:  84%|########4 | 252/300 [07:18<01:22,  1.73s/it]
2024-08-07 19:11:22,955 - INFO: train loss: 0.60:  85%|########5 | 255/300 [07:23<01:18,  1.75s/it]
2024-08-07 19:11:27,660 - INFO: train loss: 0.84:  86%|########6 | 258/300 [07:28<01:11,  1.69s/it]
2024-08-07 19:11:32,389 - INFO: train loss: 0.86:  87%|########7 | 261/300 [07:33<01:04,  1.66s/it]
2024-08-07 19:11:37,357 - INFO: train loss: 0.81:  88%|########8 | 264/300 [07:37<00:59,  1.66s/it]
2024-08-07 19:11:42,244 - INFO: train loss: 0.65:  89%|########9 | 267/300 [07:42<00:54,  1.65s/it]
2024-08-07 19:11:47,713 - INFO: train loss: 0.72:  90%|######### | 270/300 [07:48<00:51,  1.70s/it]
2024-08-07 19:11:52,431 - INFO: train loss: 0.78:  91%|#########1| 273/300 [07:53<00:44,  1.66s/it]
2024-08-07 19:11:57,157 - INFO: train loss: 0.96:  92%|#########2| 276/300 [07:57<00:39,  1.64s/it]
2024-08-07 19:12:02,221 - INFO: train loss: 0.81:  93%|#########3| 279/300 [08:02<00:34,  1.65s/it]
2024-08-07 19:12:07,076 - INFO: train loss: 0.86:  94%|#########3| 282/300 [08:07<00:29,  1.64s/it]
2024-08-07 19:12:12,287 - INFO: train loss: 0.73:  95%|#########5| 285/300 [08:12<00:25,  1.67s/it]
2024-08-07 19:12:17,309 - INFO: train loss: 0.91:  96%|#########6| 288/300 [08:17<00:20,  1.67s/it]
2024-08-07 19:12:22,028 - INFO: train loss: 1.20:  97%|#########7| 291/300 [08:22<00:14,  1.64s/it]
2024-08-07 19:12:27,110 - INFO: train loss: 1.26:  98%|#########8| 294/300 [08:27<00:09,  1.66s/it]
2024-08-07 19:12:31,838 - INFO: train loss: 1.01:  99%|#########9| 297/300 [08:32<00:04,  1.63s/it]
2024-08-07 19:12:37,289 - INFO: train loss: 0.93: 100%|##########| 300/300 [08:37<00:00,  1.69s/it]
2024-08-07 19:12:37,289 - INFO: train loss: 0.93: 100%|##########| 300/300 [08:37<00:00,  1.73s/it]
2024-08-07 19:12:37,289 - INFO: Saving last model checkpoint to /app/output
2024-08-07 19:12:37,289 - INFO: Saving checkpoint..
2024-08-07 19:13:03,140 - INFO: Starting validation inference
2024-08-07 19:13:03,141 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 19:13:08,314 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:04,  1.72s/it]
2024-08-07 19:13:09,837 - INFO: validation progress:   8%|8         | 6/75 [00:06<01:09,  1.01s/it]
2024-08-07 19:13:11,269 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:50,  1.31it/s]
2024-08-07 19:13:12,770 - INFO: validation progress:  16%|#6        | 12/75 [00:09<00:41,  1.51it/s]
2024-08-07 19:13:14,375 - INFO: validation progress:  20%|##        | 15/75 [00:11<00:36,  1.62it/s]
2024-08-07 19:13:16,027 - INFO: validation progress:  24%|##4       | 18/75 [00:12<00:33,  1.68it/s]
2024-08-07 19:13:17,755 - INFO: validation progress:  28%|##8       | 21/75 [00:14<00:31,  1.70it/s]
2024-08-07 19:13:19,293 - INFO: validation progress:  32%|###2      | 24/75 [00:16<00:28,  1.77it/s]
2024-08-07 19:13:20,991 - INFO: validation progress:  36%|###6      | 27/75 [00:17<00:27,  1.77it/s]
2024-08-07 19:13:22,668 - INFO: validation progress:  40%|####      | 30/75 [00:19<00:25,  1.78it/s]
2024-08-07 19:13:24,299 - INFO: validation progress:  44%|####4     | 33/75 [00:21<00:23,  1.80it/s]
2024-08-07 19:13:25,932 - INFO: validation progress:  48%|####8     | 36/75 [00:22<00:21,  1.81it/s]
2024-08-07 19:13:27,679 - INFO: validation progress:  52%|#####2    | 39/75 [00:24<00:20,  1.78it/s]
2024-08-07 19:13:29,263 - INFO: validation progress:  56%|#####6    | 42/75 [00:26<00:18,  1.81it/s]
2024-08-07 19:13:30,858 - INFO: validation progress:  60%|######    | 45/75 [00:27<00:16,  1.83it/s]
2024-08-07 19:13:32,367 - INFO: validation progress:  64%|######4   | 48/75 [00:29<00:14,  1.88it/s]
2024-08-07 19:13:33,874 - INFO: validation progress:  68%|######8   | 51/75 [00:30<00:12,  1.91it/s]
2024-08-07 19:13:35,532 - INFO: validation progress:  72%|#######2  | 54/75 [00:32<00:11,  1.88it/s]
2024-08-07 19:13:37,126 - INFO: validation progress:  76%|#######6  | 57/75 [00:33<00:09,  1.88it/s]
2024-08-07 19:13:38,658 - INFO: validation progress:  80%|########  | 60/75 [00:35<00:07,  1.90it/s]
2024-08-07 19:13:40,459 - INFO: validation progress:  84%|########4 | 63/75 [00:37<00:06,  1.82it/s]
2024-08-07 19:13:42,139 - INFO: validation progress:  88%|########8 | 66/75 [00:38<00:04,  1.81it/s]
2024-08-07 19:13:43,821 - INFO: validation progress:  92%|#########2| 69/75 [00:40<00:03,  1.80it/s]
2024-08-07 19:13:45,282 - INFO: validation progress:  96%|#########6| 72/75 [00:42<00:01,  1.87it/s]
2024-08-07 19:13:47,023 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.82it/s]
2024-08-07 19:13:47,026 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.71it/s]
2024-08-07 19:13:47,250 - INFO: Validation AUC: 0.95527
2024-08-07 19:13:47,251 - INFO: Mean validation loss: 0.80791
2024-08-07 19:13:47,370 - INFO: Training Epoch: 2 / 10
2024-08-07 19:13:47,370 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 19:13:50,662 - INFO: Evaluation step: 300
2024-08-07 19:13:55,857 - INFO: train loss: 0.61:   1%|1         | 3/300 [00:08<14:00,  2.83s/it]
2024-08-07 19:14:00,671 - INFO: train loss: 0.58:   2%|2         | 6/300 [00:13<10:19,  2.11s/it]
2024-08-07 19:14:05,790 - INFO: train loss: 0.55:   3%|3         | 9/300 [00:18<09:20,  1.92s/it]
2024-08-07 19:14:10,424 - INFO: train loss: 0.59:   4%|4         | 12/300 [00:23<08:31,  1.77s/it]
2024-08-07 19:14:15,917 - INFO: train loss: 0.58:   5%|5         | 15/300 [00:28<08:31,  1.80s/it]
2024-08-07 19:14:20,795 - INFO: train loss: 0.53:   6%|6         | 18/300 [00:33<08:09,  1.74s/it]
2024-08-07 19:14:25,501 - INFO: train loss: 0.53:   7%|7         | 21/300 [00:38<07:49,  1.68s/it]
2024-08-07 19:14:30,895 - INFO: train loss: 0.46:   8%|8         | 24/300 [00:43<07:54,  1.72s/it]
2024-08-07 19:14:35,892 - INFO: train loss: 0.72:   9%|9         | 27/300 [00:48<07:44,  1.70s/it]
2024-08-07 19:14:41,067 - INFO: train loss: 1.49:  10%|#         | 30/300 [00:53<07:41,  1.71s/it]
2024-08-07 19:14:46,191 - INFO: train loss: 1.54:  11%|#1        | 33/300 [00:58<07:36,  1.71s/it]
2024-08-07 19:14:51,170 - INFO: train loss: 1.49:  12%|#2        | 36/300 [01:03<07:27,  1.69s/it]
2024-08-07 19:14:56,391 - INFO: train loss: 0.86:  13%|#3        | 39/300 [01:09<07:25,  1.71s/it]
2024-08-07 19:15:01,709 - INFO: train loss: 0.94:  14%|#4        | 42/300 [01:14<07:25,  1.73s/it]
2024-08-07 19:15:06,886 - INFO: train loss: 0.91:  15%|#5        | 45/300 [01:19<07:20,  1.73s/it]
2024-08-07 19:15:11,549 - INFO: train loss: 0.86:  16%|#6        | 48/300 [01:24<07:02,  1.67s/it]
2024-08-07 19:15:16,858 - INFO: train loss: 0.63:  17%|#7        | 51/300 [01:29<07:04,  1.70s/it]
2024-08-07 19:15:22,131 - INFO: train loss: 0.70:  18%|#8        | 54/300 [01:34<07:03,  1.72s/it]
2024-08-07 19:15:27,297 - INFO: train loss: 1.06:  19%|#9        | 57/300 [01:39<06:58,  1.72s/it]
2024-08-07 19:15:32,362 - INFO: train loss: 1.05:  20%|##        | 60/300 [01:44<06:50,  1.71s/it]
2024-08-07 19:15:37,259 - INFO: train loss: 1.12:  21%|##1       | 63/300 [01:49<06:39,  1.69s/it]
2024-08-07 19:15:42,011 - INFO: train loss: 0.78:  22%|##2       | 66/300 [01:54<06:27,  1.66s/it]
2024-08-07 19:15:47,220 - INFO: train loss: 0.55:  23%|##3       | 69/300 [01:59<06:28,  1.68s/it]
2024-08-07 19:15:52,099 - INFO: train loss: 0.51:  24%|##4       | 72/300 [02:04<06:19,  1.66s/it]
2024-08-07 19:15:57,267 - INFO: train loss: 0.65:  25%|##5       | 75/300 [02:09<06:18,  1.68s/it]
2024-08-07 19:16:02,338 - INFO: train loss: 0.74:  26%|##6       | 78/300 [02:14<06:13,  1.68s/it]
2024-08-07 19:16:07,875 - INFO: train loss: 0.88:  27%|##7       | 81/300 [02:20<06:19,  1.73s/it]
2024-08-07 19:16:13,320 - INFO: train loss: 0.74:  28%|##8       | 84/300 [02:25<06:19,  1.76s/it]
2024-08-07 19:16:18,154 - INFO: train loss: 0.53:  29%|##9       | 87/300 [02:30<06:04,  1.71s/it]
2024-08-07 19:16:24,044 - INFO: train loss: 0.39:  30%|###       | 90/300 [02:36<06:15,  1.79s/it]
2024-08-07 19:16:29,104 - INFO: train loss: 0.34:  31%|###1      | 93/300 [02:41<06:03,  1.76s/it]
2024-08-07 19:16:34,700 - INFO: train loss: 0.41:  32%|###2      | 96/300 [02:47<06:05,  1.79s/it]
2024-08-07 19:16:40,060 - INFO: train loss: 0.44:  33%|###3      | 99/300 [02:52<05:59,  1.79s/it]
2024-08-07 19:16:45,111 - INFO: train loss: 0.53:  34%|###4      | 102/300 [02:57<05:47,  1.76s/it]
2024-08-07 19:16:51,649 - INFO: train loss: 0.51:  35%|###5      | 105/300 [03:04<06:07,  1.88s/it]
2024-08-07 19:16:57,221 - INFO: train loss: 0.45:  36%|###6      | 108/300 [03:09<06:00,  1.88s/it]
2024-08-07 19:17:01,876 - INFO: train loss: 0.43:  37%|###7      | 111/300 [03:14<05:36,  1.78s/it]
2024-08-07 19:17:06,879 - INFO: train loss: 0.40:  38%|###8      | 114/300 [03:19<05:24,  1.75s/it]
2024-08-07 19:17:11,988 - INFO: train loss: 0.43:  39%|###9      | 117/300 [03:24<05:17,  1.73s/it]
2024-08-07 19:17:16,949 - INFO: train loss: 0.36:  40%|####      | 120/300 [03:29<05:07,  1.71s/it]
2024-08-07 19:17:21,540 - INFO: train loss: 0.51:  41%|####1     | 123/300 [03:34<04:53,  1.66s/it]
2024-08-07 19:17:26,517 - INFO: train loss: 0.58:  42%|####2     | 126/300 [03:39<04:48,  1.66s/it]
2024-08-07 19:17:31,848 - INFO: train loss: 0.62:  43%|####3     | 129/300 [03:44<04:49,  1.69s/it]
2024-08-07 19:17:36,859 - INFO: train loss: 0.69:  44%|####4     | 132/300 [03:49<04:43,  1.69s/it]
2024-08-07 19:17:42,055 - INFO: train loss: 0.55:  45%|####5     | 135/300 [03:54<04:40,  1.70s/it]
2024-08-07 19:17:46,785 - INFO: train loss: 0.59:  46%|####6     | 138/300 [03:59<04:29,  1.66s/it]
2024-08-07 19:17:51,609 - INFO: train loss: 0.59:  47%|####6     | 141/300 [04:04<04:21,  1.65s/it]
2024-08-07 19:17:56,419 - INFO: train loss: 0.59:  48%|####8     | 144/300 [04:09<04:14,  1.63s/it]
2024-08-07 19:18:01,455 - INFO: train loss: 0.58:  49%|####9     | 147/300 [04:14<04:11,  1.65s/it]
2024-08-07 19:18:06,183 - INFO: train loss: 0.59:  50%|#####     | 150/300 [04:18<04:03,  1.63s/it]
2024-08-07 19:18:11,341 - INFO: train loss: 0.61:  51%|#####1    | 153/300 [04:23<04:03,  1.65s/it]
2024-08-07 19:18:16,482 - INFO: train loss: 0.60:  52%|#####2    | 156/300 [04:29<04:00,  1.67s/it]
2024-08-07 19:18:21,284 - INFO: train loss: 0.71:  53%|#####3    | 159/300 [04:33<03:52,  1.65s/it]
2024-08-07 19:18:26,276 - INFO: train loss: 0.68:  54%|#####4    | 162/300 [04:38<03:48,  1.65s/it]
2024-08-07 19:18:31,378 - INFO: train loss: 0.76:  55%|#####5    | 165/300 [04:44<03:45,  1.67s/it]
2024-08-07 19:18:36,552 - INFO: train loss: 0.64:  56%|#####6    | 168/300 [04:49<03:42,  1.69s/it]
2024-08-07 19:18:41,487 - INFO: train loss: 0.63:  57%|#####6    | 171/300 [04:54<03:35,  1.67s/it]
2024-08-07 19:18:46,411 - INFO: train loss: 0.54:  58%|#####8    | 174/300 [04:59<03:29,  1.66s/it]
2024-08-07 19:18:51,024 - INFO: train loss: 0.48:  59%|#####8    | 177/300 [05:03<03:19,  1.63s/it]
2024-08-07 19:18:56,135 - INFO: train loss: 0.47:  60%|######    | 180/300 [05:08<03:17,  1.65s/it]
2024-08-07 19:19:00,934 - INFO: train loss: 0.48:  61%|######1   | 183/300 [05:13<03:11,  1.63s/it]
2024-08-07 19:19:06,044 - INFO: train loss: 0.53:  62%|######2   | 186/300 [05:18<03:08,  1.65s/it]
2024-08-07 19:19:10,627 - INFO: train loss: 0.67:  63%|######3   | 189/300 [05:23<02:59,  1.62s/it]
2024-08-07 19:19:15,307 - INFO: train loss: 0.62:  64%|######4   | 192/300 [05:27<02:52,  1.60s/it]
2024-08-07 19:19:20,224 - INFO: train loss: 0.58:  65%|######5   | 195/300 [05:32<02:49,  1.61s/it]
2024-08-07 19:19:25,369 - INFO: train loss: 0.49:  66%|######6   | 198/300 [05:37<02:47,  1.64s/it]
2024-08-07 19:19:30,555 - INFO: train loss: 0.42:  67%|######7   | 201/300 [05:43<02:45,  1.67s/it]
2024-08-07 19:19:35,560 - INFO: train loss: 0.53:  68%|######8   | 204/300 [05:48<02:40,  1.67s/it]
2024-08-07 19:19:40,911 - INFO: train loss: 0.60:  69%|######9   | 207/300 [05:53<02:38,  1.70s/it]
2024-08-07 19:19:45,894 - INFO: train loss: 0.73:  70%|#######   | 210/300 [05:58<02:32,  1.69s/it]
2024-08-07 19:19:50,909 - INFO: train loss: 0.70:  71%|#######1  | 213/300 [06:03<02:26,  1.68s/it]
2024-08-07 19:19:55,567 - INFO: train loss: 0.72:  72%|#######2  | 216/300 [06:08<02:18,  1.65s/it]
2024-08-07 19:20:00,750 - INFO: train loss: 0.71:  73%|#######3  | 219/300 [06:13<02:15,  1.67s/it]
2024-08-07 19:20:05,857 - INFO: train loss: 0.61:  74%|#######4  | 222/300 [06:18<02:11,  1.68s/it]
2024-08-07 19:20:10,434 - INFO: train loss: 0.60:  75%|#######5  | 225/300 [06:23<02:02,  1.63s/it]
2024-08-07 19:20:15,355 - INFO: train loss: 0.54:  76%|#######6  | 228/300 [06:27<01:57,  1.64s/it]
2024-08-07 19:20:21,092 - INFO: train loss: 0.52:  77%|#######7  | 231/300 [06:33<01:58,  1.72s/it]
2024-08-07 19:20:26,018 - INFO: train loss: 0.45:  78%|#######8  | 234/300 [06:38<01:51,  1.70s/it]
2024-08-07 19:20:31,252 - INFO: train loss: 0.60:  79%|#######9  | 237/300 [06:43<01:47,  1.71s/it]
2024-08-07 19:20:36,386 - INFO: train loss: 0.76:  80%|########  | 240/300 [06:49<01:42,  1.71s/it]
2024-08-07 19:20:41,776 - INFO: train loss: 0.72:  81%|########1 | 243/300 [06:54<01:38,  1.74s/it]
2024-08-07 19:20:46,915 - INFO: train loss: 0.82:  82%|########2 | 246/300 [06:59<01:33,  1.73s/it]
2024-08-07 19:20:51,388 - INFO: train loss: 0.44:  83%|########2 | 249/300 [07:04<01:24,  1.66s/it]
2024-08-07 19:20:56,415 - INFO: train loss: 0.68:  84%|########4 | 252/300 [07:09<01:19,  1.66s/it]
2024-08-07 19:21:01,275 - INFO: train loss: 0.78:  85%|########5 | 255/300 [07:13<01:14,  1.65s/it]
2024-08-07 19:21:06,486 - INFO: train loss: 0.77:  86%|########6 | 258/300 [07:19<01:10,  1.68s/it]
2024-08-07 19:21:11,351 - INFO: train loss: 0.71:  87%|########7 | 261/300 [07:23<01:04,  1.66s/it]
2024-08-07 19:21:16,069 - INFO: train loss: 0.46:  88%|########8 | 264/300 [07:28<00:58,  1.63s/it]
2024-08-07 19:21:20,879 - INFO: train loss: 0.46:  89%|########9 | 267/300 [07:33<00:53,  1.62s/it]
2024-08-07 19:21:26,072 - INFO: train loss: 0.50:  90%|######### | 270/300 [07:38<00:49,  1.66s/it]
2024-08-07 19:21:31,164 - INFO: train loss: 0.54:  91%|#########1| 273/300 [07:43<00:45,  1.67s/it]
2024-08-07 19:21:35,863 - INFO: train loss: 0.80:  92%|#########2| 276/300 [07:48<00:39,  1.64s/it]
2024-08-07 19:21:40,944 - INFO: train loss: 0.74:  93%|#########3| 279/300 [07:53<00:34,  1.65s/it]
2024-08-07 19:21:45,764 - INFO: train loss: 0.86:  94%|#########3| 282/300 [07:58<00:29,  1.64s/it]
2024-08-07 19:21:50,668 - INFO: train loss: 0.61:  95%|#########5| 285/300 [08:03<00:24,  1.64s/it]
2024-08-07 19:21:55,646 - INFO: train loss: 0.85:  96%|#########6| 288/300 [08:08<00:19,  1.64s/it]
2024-08-07 19:22:00,327 - INFO: train loss: 0.79:  97%|#########7| 291/300 [08:12<00:14,  1.62s/it]
2024-08-07 19:22:04,848 - INFO: train loss: 0.76:  98%|#########8| 294/300 [08:17<00:09,  1.59s/it]
2024-08-07 19:22:09,450 - INFO: train loss: 0.51:  99%|#########9| 297/300 [08:22<00:04,  1.57s/it]
2024-08-07 19:22:14,517 - INFO: train loss: 0.74: 100%|##########| 300/300 [08:27<00:00,  1.61s/it]
2024-08-07 19:22:14,517 - INFO: train loss: 0.74: 100%|##########| 300/300 [08:27<00:00,  1.69s/it]
2024-08-07 19:22:14,517 - INFO: Saving last model checkpoint to /app/output
2024-08-07 19:22:14,518 - INFO: Saving checkpoint..
2024-08-07 19:22:40,850 - INFO: Starting validation inference
2024-08-07 19:22:40,851 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 19:22:46,789 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:22,  1.98s/it]
2024-08-07 19:22:48,304 - INFO: validation progress:   8%|8         | 6/75 [00:07<01:16,  1.11s/it]
2024-08-07 19:22:49,749 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:54,  1.21it/s]
2024-08-07 19:22:51,249 - INFO: validation progress:  16%|#6        | 12/75 [00:10<00:43,  1.44it/s]
2024-08-07 19:22:52,858 - INFO: validation progress:  20%|##        | 15/75 [00:12<00:38,  1.57it/s]
2024-08-07 19:22:54,468 - INFO: validation progress:  24%|##4       | 18/75 [00:13<00:34,  1.66it/s]
2024-08-07 19:22:56,098 - INFO: validation progress:  28%|##8       | 21/75 [00:15<00:31,  1.71it/s]
2024-08-07 19:22:57,589 - INFO: validation progress:  32%|###2      | 24/75 [00:16<00:28,  1.80it/s]
2024-08-07 19:22:59,293 - INFO: validation progress:  36%|###6      | 27/75 [00:18<00:26,  1.79it/s]
2024-08-07 19:23:00,869 - INFO: validation progress:  40%|####      | 30/75 [00:20<00:24,  1.82it/s]
2024-08-07 19:23:02,505 - INFO: validation progress:  44%|####4     | 33/75 [00:21<00:23,  1.82it/s]
2024-08-07 19:23:04,190 - INFO: validation progress:  48%|####8     | 36/75 [00:23<00:21,  1.81it/s]
2024-08-07 19:23:05,909 - INFO: validation progress:  52%|#####2    | 39/75 [00:25<00:20,  1.79it/s]
2024-08-07 19:23:07,486 - INFO: validation progress:  56%|#####6    | 42/75 [00:26<00:18,  1.82it/s]
2024-08-07 19:23:09,034 - INFO: validation progress:  60%|######    | 45/75 [00:28<00:16,  1.86it/s]
2024-08-07 19:23:10,493 - INFO: validation progress:  64%|######4   | 48/75 [00:29<00:14,  1.91it/s]
2024-08-07 19:23:12,006 - INFO: validation progress:  68%|######8   | 51/75 [00:31<00:12,  1.93it/s]
2024-08-07 19:23:13,687 - INFO: validation progress:  72%|#######2  | 54/75 [00:32<00:11,  1.89it/s]
2024-08-07 19:23:15,257 - INFO: validation progress:  76%|#######6  | 57/75 [00:34<00:09,  1.89it/s]
2024-08-07 19:23:16,698 - INFO: validation progress:  80%|########  | 60/75 [00:35<00:07,  1.95it/s]
2024-08-07 19:23:18,403 - INFO: validation progress:  84%|########4 | 63/75 [00:37<00:06,  1.89it/s]
2024-08-07 19:23:20,037 - INFO: validation progress:  88%|########8 | 66/75 [00:39<00:04,  1.87it/s]
2024-08-07 19:23:21,764 - INFO: validation progress:  92%|#########2| 69/75 [00:40<00:03,  1.83it/s]
2024-08-07 19:23:23,241 - INFO: validation progress:  96%|#########6| 72/75 [00:42<00:01,  1.89it/s]
2024-08-07 19:23:24,969 - INFO: validation progress: 100%|##########| 75/75 [00:44<00:00,  1.84it/s]
2024-08-07 19:23:24,971 - INFO: validation progress: 100%|##########| 75/75 [00:44<00:00,  1.70it/s]
2024-08-07 19:23:25,300 - INFO: Validation AUC: 0.97725
2024-08-07 19:23:25,300 - INFO: Mean validation loss: 1.16324
2024-08-07 19:23:25,706 - INFO: Training Epoch: 3 / 10
2024-08-07 19:23:25,706 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 19:23:29,575 - INFO: Evaluation step: 300
2024-08-07 19:23:34,831 - INFO: train loss: 0.59:   1%|1         | 3/300 [00:09<15:03,  3.04s/it]
2024-08-07 19:23:39,501 - INFO: train loss: 0.48:   2%|2         | 6/300 [00:13<10:37,  2.17s/it]
2024-08-07 19:23:44,342 - INFO: train loss: 0.64:   3%|3         | 9/300 [00:18<09:17,  1.91s/it]
2024-08-07 19:23:48,851 - INFO: train loss: 0.58:   4%|4         | 12/300 [00:23<08:24,  1.75s/it]
2024-08-07 19:23:54,092 - INFO: train loss: 0.52:   5%|5         | 15/300 [00:28<08:18,  1.75s/it]
2024-08-07 19:23:58,855 - INFO: train loss: 0.50:   6%|6         | 18/300 [00:33<07:58,  1.70s/it]
2024-08-07 19:24:03,561 - INFO: train loss: 0.52:   7%|7         | 21/300 [00:37<07:41,  1.65s/it]
2024-08-07 19:24:08,537 - INFO: train loss: 0.55:   8%|8         | 24/300 [00:42<07:36,  1.66s/it]
2024-08-07 19:24:13,683 - INFO: train loss: 0.47:   9%|9         | 27/300 [00:47<07:37,  1.67s/it]
2024-08-07 19:24:19,175 - INFO: train loss: 0.44:  10%|#         | 30/300 [00:53<07:45,  1.72s/it]
2024-08-07 19:24:24,253 - INFO: train loss: 0.50:  11%|#1        | 33/300 [00:58<07:37,  1.71s/it]
2024-08-07 19:24:29,065 - INFO: train loss: 0.50:  12%|#2        | 36/300 [01:03<07:23,  1.68s/it]
2024-08-07 19:24:33,566 - INFO: train loss: 0.52:  13%|#3        | 39/300 [01:07<07:04,  1.63s/it]
2024-08-07 19:24:38,556 - INFO: train loss: 0.33:  14%|#4        | 42/300 [01:12<07:02,  1.64s/it]
2024-08-07 19:24:43,747 - INFO: train loss: 0.52:  15%|#5        | 45/300 [01:18<07:04,  1.67s/it]
2024-08-07 19:24:48,683 - INFO: train loss: 0.44:  16%|#6        | 48/300 [01:22<06:58,  1.66s/it]
2024-08-07 19:24:53,798 - INFO: train loss: 0.48:  17%|#7        | 51/300 [01:28<06:56,  1.67s/it]
2024-08-07 19:24:59,069 - INFO: train loss: 0.38:  18%|#8        | 54/300 [01:33<06:57,  1.70s/it]
2024-08-07 19:25:04,460 - INFO: train loss: 0.38:  19%|#9        | 57/300 [01:38<06:59,  1.73s/it]
2024-08-07 19:25:09,478 - INFO: train loss: 0.38:  20%|##        | 60/300 [01:43<06:50,  1.71s/it]
2024-08-07 19:25:14,373 - INFO: train loss: 0.44:  21%|##1       | 63/300 [01:48<06:39,  1.69s/it]
2024-08-07 19:25:19,465 - INFO: train loss: 0.76:  22%|##2       | 66/300 [01:53<06:35,  1.69s/it]
2024-08-07 19:25:24,162 - INFO: train loss: 0.80:  23%|##3       | 69/300 [01:58<06:21,  1.65s/it]
2024-08-07 19:25:29,792 - INFO: train loss: 1.08:  24%|##4       | 72/300 [02:04<06:32,  1.72s/it]
2024-08-07 19:25:34,796 - INFO: train loss: 0.87:  25%|##5       | 75/300 [02:09<06:23,  1.70s/it]
2024-08-07 19:25:40,054 - INFO: train loss: 0.74:  26%|##6       | 78/300 [02:14<06:21,  1.72s/it]
2024-08-07 19:25:45,549 - INFO: train loss: 0.46:  27%|##7       | 81/300 [02:19<06:23,  1.75s/it]
2024-08-07 19:25:50,736 - INFO: train loss: 0.35:  28%|##8       | 84/300 [02:25<06:17,  1.75s/it]
2024-08-07 19:25:55,329 - INFO: train loss: 0.52:  29%|##9       | 87/300 [02:29<05:58,  1.68s/it]
2024-08-07 19:26:00,146 - INFO: train loss: 0.53:  30%|###       | 90/300 [02:34<05:48,  1.66s/it]
2024-08-07 19:26:05,185 - INFO: train loss: 0.45:  31%|###1      | 93/300 [02:39<05:44,  1.66s/it]
2024-08-07 19:26:10,016 - INFO: train loss: 0.61:  32%|###2      | 96/300 [02:44<05:36,  1.65s/it]
2024-08-07 19:26:14,846 - INFO: train loss: 0.41:  33%|###3      | 99/300 [02:49<05:29,  1.64s/it]
2024-08-07 19:26:20,159 - INFO: train loss: 0.49:  34%|###4      | 102/300 [02:54<05:32,  1.68s/it]
2024-08-07 19:26:25,375 - INFO: train loss: 0.30:  35%|###5      | 105/300 [02:59<05:30,  1.70s/it]
2024-08-07 19:26:30,313 - INFO: train loss: 0.41:  36%|###6      | 108/300 [03:04<05:22,  1.68s/it]
2024-08-07 19:26:36,208 - INFO: train loss: 0.33:  37%|###7      | 111/300 [03:10<05:33,  1.77s/it]
2024-08-07 19:26:40,662 - INFO: train loss: 0.48:  38%|###8      | 114/300 [03:14<05:12,  1.68s/it]
2024-08-07 19:26:45,404 - INFO: train loss: 0.50:  39%|###9      | 117/300 [03:19<05:02,  1.65s/it]
2024-08-07 19:26:51,145 - INFO: train loss: 0.54:  40%|####      | 120/300 [03:25<05:11,  1.73s/it]
2024-08-07 19:26:56,365 - INFO: train loss: 0.56:  41%|####1     | 123/300 [03:30<05:06,  1.73s/it]
2024-08-07 19:27:01,889 - INFO: train loss: 0.53:  42%|####2     | 126/300 [03:36<05:07,  1.77s/it]
2024-08-07 19:27:06,744 - INFO: train loss: 0.44:  43%|####3     | 129/300 [03:41<04:54,  1.72s/it]
2024-08-07 19:27:11,914 - INFO: train loss: 0.45:  44%|####4     | 132/300 [03:46<04:49,  1.72s/it]
2024-08-07 19:27:16,646 - INFO: train loss: 0.29:  45%|####5     | 135/300 [03:50<04:36,  1.68s/it]
2024-08-07 19:27:21,536 - INFO: train loss: 0.30:  46%|####6     | 138/300 [03:55<04:29,  1.66s/it]
2024-08-07 19:27:27,003 - INFO: train loss: 0.31:  47%|####6     | 141/300 [04:01<04:32,  1.71s/it]
2024-08-07 19:27:32,177 - INFO: train loss: 0.27:  48%|####8     | 144/300 [04:06<04:27,  1.72s/it]
2024-08-07 19:27:37,259 - INFO: train loss: 0.22:  49%|####9     | 147/300 [04:11<04:21,  1.71s/it]
2024-08-07 19:27:42,604 - INFO: train loss: 0.23:  50%|#####     | 150/300 [04:16<04:19,  1.73s/it]
2024-08-07 19:27:47,759 - INFO: train loss: 0.25:  51%|#####1    | 153/300 [04:22<04:13,  1.73s/it]
2024-08-07 19:27:52,999 - INFO: train loss: 0.27:  52%|#####2    | 156/300 [04:27<04:09,  1.73s/it]
2024-08-07 19:27:58,082 - INFO: train loss: 0.30:  53%|#####3    | 159/300 [04:32<04:02,  1.72s/it]
2024-08-07 19:28:03,115 - INFO: train loss: 0.36:  54%|#####4    | 162/300 [04:37<03:55,  1.71s/it]
2024-08-07 19:28:08,035 - INFO: train loss: 0.36:  55%|#####5    | 165/300 [04:42<03:47,  1.69s/it]
2024-08-07 19:28:12,639 - INFO: train loss: 0.39:  56%|#####6    | 168/300 [04:46<03:36,  1.64s/it]
2024-08-07 19:28:17,843 - INFO: train loss: 0.38:  57%|#####6    | 171/300 [04:52<03:35,  1.67s/it]
2024-08-07 19:28:23,175 - INFO: train loss: 0.38:  58%|#####8    | 174/300 [04:57<03:34,  1.70s/it]
2024-08-07 19:28:27,933 - INFO: train loss: 0.40:  59%|#####8    | 177/300 [05:02<03:25,  1.67s/it]
2024-08-07 19:28:33,051 - INFO: train loss: 0.40:  60%|######    | 180/300 [05:07<03:21,  1.68s/it]
2024-08-07 19:28:38,074 - INFO: train loss: 0.42:  61%|######1   | 183/300 [05:12<03:16,  1.68s/it]
2024-08-07 19:28:42,833 - INFO: train loss: 0.34:  62%|######2   | 186/300 [05:17<03:08,  1.65s/it]
2024-08-07 19:28:47,430 - INFO: train loss: 0.34:  63%|######3   | 189/300 [05:21<02:59,  1.61s/it]
2024-08-07 19:28:52,617 - INFO: train loss: 0.34:  64%|######4   | 192/300 [05:26<02:58,  1.65s/it]
2024-08-07 19:28:57,845 - INFO: train loss: 0.41:  65%|######5   | 195/300 [05:32<02:56,  1.68s/it]
2024-08-07 19:29:02,658 - INFO: train loss: 0.49:  66%|######6   | 198/300 [05:36<02:48,  1.66s/it]
2024-08-07 19:29:08,210 - INFO: train loss: 0.49:  67%|######7   | 201/300 [05:42<02:49,  1.71s/it]
2024-08-07 19:29:12,848 - INFO: train loss: 0.40:  68%|######8   | 204/300 [05:47<02:39,  1.66s/it]
2024-08-07 19:29:17,976 - INFO: train loss: 0.35:  69%|######9   | 207/300 [05:52<02:35,  1.68s/it]
2024-08-07 19:29:22,891 - INFO: train loss: 0.37:  70%|#######   | 210/300 [05:57<02:29,  1.67s/it]
2024-08-07 19:29:27,636 - INFO: train loss: 0.46:  71%|#######1  | 213/300 [06:01<02:22,  1.64s/it]
2024-08-07 19:29:32,488 - INFO: train loss: 0.46:  72%|#######2  | 216/300 [06:06<02:17,  1.63s/it]
2024-08-07 19:29:37,637 - INFO: train loss: 0.54:  73%|#######3  | 219/300 [06:11<02:14,  1.66s/it]
2024-08-07 19:29:42,650 - INFO: train loss: 0.39:  74%|#######4  | 222/300 [06:16<02:09,  1.66s/it]
2024-08-07 19:29:48,009 - INFO: train loss: 0.40:  75%|#######5  | 225/300 [06:22<02:07,  1.70s/it]
2024-08-07 19:29:53,119 - INFO: train loss: 0.39:  76%|#######6  | 228/300 [06:27<02:02,  1.70s/it]
2024-08-07 19:29:57,838 - INFO: train loss: 0.35:  77%|#######7  | 231/300 [06:32<01:54,  1.66s/it]
2024-08-07 19:30:02,931 - INFO: train loss: 0.32:  78%|#######8  | 234/300 [06:37<01:50,  1.67s/it]
2024-08-07 19:30:07,740 - INFO: train loss: 0.40:  79%|#######9  | 237/300 [06:42<01:44,  1.65s/it]
2024-08-07 19:30:12,602 - INFO: train loss: 0.53:  80%|########  | 240/300 [06:46<01:38,  1.64s/it]
2024-08-07 19:30:17,604 - INFO: train loss: 0.58:  81%|########1 | 243/300 [06:51<01:34,  1.65s/it]
2024-08-07 19:30:22,725 - INFO: train loss: 0.49:  82%|########2 | 246/300 [06:57<01:30,  1.67s/it]
2024-08-07 19:30:27,864 - INFO: train loss: 0.30:  83%|########2 | 249/300 [07:02<01:25,  1.68s/it]
2024-08-07 19:30:34,111 - INFO: train loss: 0.28:  84%|########4 | 252/300 [07:08<01:26,  1.80s/it]
2024-08-07 19:30:39,163 - INFO: train loss: 0.27:  85%|########5 | 255/300 [07:13<01:19,  1.77s/it]
2024-08-07 19:30:44,370 - INFO: train loss: 0.33:  86%|########6 | 258/300 [07:18<01:13,  1.76s/it]
2024-08-07 19:30:49,655 - INFO: train loss: 0.32:  87%|########7 | 261/300 [07:23<01:08,  1.76s/it]
2024-08-07 19:30:54,837 - INFO: train loss: 0.35:  88%|########8 | 264/300 [07:29<01:02,  1.75s/it]
2024-08-07 19:30:59,962 - INFO: train loss: 0.38:  89%|########9 | 267/300 [07:34<00:57,  1.74s/it]
2024-08-07 19:31:04,838 - INFO: train loss: 0.33:  90%|######### | 270/300 [07:39<00:51,  1.70s/it]
2024-08-07 19:31:09,868 - INFO: train loss: 0.34:  91%|#########1| 273/300 [07:44<00:45,  1.70s/it]
2024-08-07 19:31:15,082 - INFO: train loss: 0.36:  92%|#########2| 276/300 [07:49<00:40,  1.71s/it]
2024-08-07 19:31:19,851 - INFO: train loss: 0.35:  93%|#########3| 279/300 [07:54<00:35,  1.67s/it]
2024-08-07 19:31:25,179 - INFO: train loss: 0.40:  94%|#########3| 282/300 [07:59<00:30,  1.70s/it]
2024-08-07 19:31:30,292 - INFO: train loss: 0.44:  95%|#########5| 285/300 [08:04<00:25,  1.70s/it]
2024-08-07 19:31:34,977 - INFO: train loss: 0.35:  96%|#########6| 288/300 [08:09<00:19,  1.66s/it]
2024-08-07 19:31:40,691 - INFO: train loss: 0.29:  97%|#########7| 291/300 [08:14<00:15,  1.73s/it]
2024-08-07 19:31:46,068 - INFO: train loss: 0.37:  98%|#########8| 294/300 [08:20<00:10,  1.75s/it]
2024-08-07 19:31:50,951 - INFO: train loss: 0.38:  99%|#########9| 297/300 [08:25<00:05,  1.71s/it]
2024-08-07 19:31:55,957 - INFO: train loss: 0.47: 100%|##########| 300/300 [08:30<00:00,  1.70s/it]
2024-08-07 19:31:55,957 - INFO: train loss: 0.47: 100%|##########| 300/300 [08:30<00:00,  1.70s/it]
2024-08-07 19:31:55,957 - INFO: Saving last model checkpoint to /app/output
2024-08-07 19:31:55,957 - INFO: Saving checkpoint..
2024-08-07 19:32:27,078 - INFO: Starting validation inference
2024-08-07 19:32:27,079 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 19:32:32,655 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:13,  1.86s/it]
2024-08-07 19:32:34,219 - INFO: validation progress:   8%|8         | 6/75 [00:07<01:13,  1.07s/it]
2024-08-07 19:32:35,650 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:52,  1.25it/s]
2024-08-07 19:32:37,165 - INFO: validation progress:  16%|#6        | 12/75 [00:10<00:43,  1.46it/s]
2024-08-07 19:32:38,769 - INFO: validation progress:  20%|##        | 15/75 [00:11<00:37,  1.59it/s]
2024-08-07 19:32:40,325 - INFO: validation progress:  24%|##4       | 18/75 [00:13<00:33,  1.69it/s]
2024-08-07 19:32:41,955 - INFO: validation progress:  28%|##8       | 21/75 [00:14<00:31,  1.74it/s]
2024-08-07 19:32:43,448 - INFO: validation progress:  32%|###2      | 24/75 [00:16<00:28,  1.81it/s]
2024-08-07 19:32:45,091 - INFO: validation progress:  36%|###6      | 27/75 [00:18<00:26,  1.82it/s]
2024-08-07 19:32:46,670 - INFO: validation progress:  40%|####      | 30/75 [00:19<00:24,  1.84it/s]
2024-08-07 19:32:48,303 - INFO: validation progress:  44%|####4     | 33/75 [00:21<00:22,  1.84it/s]
2024-08-07 19:32:49,939 - INFO: validation progress:  48%|####8     | 36/75 [00:22<00:21,  1.84it/s]
2024-08-07 19:32:51,601 - INFO: validation progress:  52%|#####2    | 39/75 [00:24<00:19,  1.83it/s]
2024-08-07 19:32:53,177 - INFO: validation progress:  56%|#####6    | 42/75 [00:26<00:17,  1.85it/s]
2024-08-07 19:32:54,726 - INFO: validation progress:  60%|######    | 45/75 [00:27<00:15,  1.88it/s]
2024-08-07 19:32:56,182 - INFO: validation progress:  64%|######4   | 48/75 [00:29<00:14,  1.93it/s]
2024-08-07 19:32:57,686 - INFO: validation progress:  68%|######8   | 51/75 [00:30<00:12,  1.95it/s]
2024-08-07 19:32:59,264 - INFO: validation progress:  72%|#######2  | 54/75 [00:32<00:10,  1.93it/s]
2024-08-07 19:33:00,842 - INFO: validation progress:  76%|#######6  | 57/75 [00:33<00:09,  1.92it/s]
2024-08-07 19:33:02,285 - INFO: validation progress:  80%|########  | 60/75 [00:35<00:07,  1.97it/s]
2024-08-07 19:33:04,031 - INFO: validation progress:  84%|########4 | 63/75 [00:36<00:06,  1.89it/s]
2024-08-07 19:33:05,661 - INFO: validation progress:  88%|########8 | 66/75 [00:38<00:04,  1.87it/s]
2024-08-07 19:33:07,338 - INFO: validation progress:  92%|#########2| 69/75 [00:40<00:03,  1.85it/s]
2024-08-07 19:33:08,800 - INFO: validation progress:  96%|#########6| 72/75 [00:41<00:01,  1.90it/s]
2024-08-07 19:33:10,543 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.84it/s]
2024-08-07 19:33:10,546 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.73it/s]
2024-08-07 19:33:10,819 - INFO: Validation AUC: 0.97624
2024-08-07 19:33:10,819 - INFO: Mean validation loss: 0.50932
2024-08-07 19:33:11,230 - INFO: Training Epoch: 4 / 10
2024-08-07 19:33:11,231 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 19:33:15,018 - INFO: Evaluation step: 300
2024-08-07 19:33:20,645 - INFO: train loss: 0.39:   1%|1         | 3/300 [00:09<15:32,  3.14s/it]
2024-08-07 19:33:25,436 - INFO: train loss: 1.11:   2%|2         | 6/300 [00:14<10:56,  2.23s/it]
2024-08-07 19:33:30,384 - INFO: train loss: 0.87:   3%|3         | 9/300 [00:19<09:31,  1.97s/it]
2024-08-07 19:33:36,617 - INFO: train loss: 0.93:   4%|4         | 12/300 [00:25<09:38,  2.01s/it]
2024-08-07 19:33:42,136 - INFO: train loss: 0.64:   5%|5         | 15/300 [00:30<09:15,  1.95s/it]
2024-08-07 19:33:47,102 - INFO: train loss: 0.46:   6%|6         | 18/300 [00:35<08:41,  1.85s/it]
2024-08-07 19:33:51,875 - INFO: train loss: 0.42:   7%|7         | 21/300 [00:40<08:12,  1.76s/it]
2024-08-07 19:33:57,004 - INFO: train loss: 0.28:   8%|8         | 24/300 [00:45<08:02,  1.75s/it]
2024-08-07 19:34:02,259 - INFO: train loss: 0.38:   9%|9         | 27/300 [00:51<07:57,  1.75s/it]
2024-08-07 19:34:06,937 - INFO: train loss: 0.35:  10%|#         | 30/300 [00:55<07:36,  1.69s/it]
2024-08-07 19:34:11,642 - INFO: train loss: 0.37:  11%|#1        | 33/300 [01:00<07:21,  1.65s/it]
2024-08-07 19:34:16,615 - INFO: train loss: 0.35:  12%|#2        | 36/300 [01:05<07:16,  1.65s/it]
2024-08-07 19:34:21,481 - INFO: train loss: 0.20:  13%|#3        | 39/300 [01:10<07:09,  1.64s/it]
2024-08-07 19:34:26,315 - INFO: train loss: 0.17:  14%|#4        | 42/300 [01:15<07:01,  1.63s/it]
2024-08-07 19:34:31,265 - INFO: train loss: 0.16:  15%|#5        | 45/300 [01:20<06:57,  1.64s/it]
2024-08-07 19:34:36,194 - INFO: train loss: 0.27:  16%|#6        | 48/300 [01:24<06:53,  1.64s/it]
2024-08-07 19:34:41,610 - INFO: train loss: 0.28:  17%|#7        | 51/300 [01:30<07:00,  1.69s/it]
2024-08-07 19:34:46,602 - INFO: train loss: 0.33:  18%|#8        | 54/300 [01:35<06:53,  1.68s/it]
2024-08-07 19:34:51,869 - INFO: train loss: 0.31:  19%|#9        | 57/300 [01:40<06:54,  1.70s/it]
2024-08-07 19:34:57,393 - INFO: train loss: 0.27:  20%|##        | 60/300 [01:46<06:58,  1.75s/it]
2024-08-07 19:35:02,286 - INFO: train loss: 0.27:  21%|##1       | 63/300 [01:51<06:45,  1.71s/it]
2024-08-07 19:35:07,319 - INFO: train loss: 0.19:  22%|##2       | 66/300 [01:56<06:38,  1.70s/it]
2024-08-07 19:35:12,464 - INFO: train loss: 0.20:  23%|##3       | 69/300 [02:01<06:33,  1.71s/it]
2024-08-07 19:35:17,863 - INFO: train loss: 0.29:  24%|##4       | 72/300 [02:06<06:35,  1.73s/it]
2024-08-07 19:35:22,551 - INFO: train loss: 0.44:  25%|##5       | 75/300 [02:11<06:18,  1.68s/it]
2024-08-07 19:35:27,155 - INFO: train loss: 0.49:  26%|##6       | 78/300 [02:15<06:03,  1.64s/it]
2024-08-07 19:35:32,257 - INFO: train loss: 0.43:  27%|##7       | 81/300 [02:21<06:02,  1.66s/it]
2024-08-07 19:35:37,455 - INFO: train loss: 0.25:  28%|##8       | 84/300 [02:26<06:02,  1.68s/it]
2024-08-07 19:35:42,698 - INFO: train loss: 0.16:  29%|##9       | 87/300 [02:31<06:02,  1.70s/it]
2024-08-07 19:35:47,073 - INFO: train loss: 0.13:  30%|###       | 90/300 [02:35<05:41,  1.63s/it]
2024-08-07 19:35:52,294 - INFO: train loss: 0.10:  31%|###1      | 93/300 [02:41<05:43,  1.66s/it]
2024-08-07 19:35:57,477 - INFO: train loss: 0.21:  32%|###2      | 96/300 [02:46<05:42,  1.68s/it]
2024-08-07 19:36:02,234 - INFO: train loss: 0.26:  33%|###3      | 99/300 [02:51<05:32,  1.65s/it]
2024-08-07 19:36:06,754 - INFO: train loss: 0.25:  34%|###4      | 102/300 [02:55<05:18,  1.61s/it]
2024-08-07 19:36:11,705 - INFO: train loss: 0.29:  35%|###5      | 105/300 [03:00<05:16,  1.62s/it]
2024-08-07 19:36:16,678 - INFO: train loss: 0.34:  36%|###6      | 108/300 [03:05<05:13,  1.63s/it]
2024-08-07 19:36:21,317 - INFO: train loss: 0.34:  37%|###7      | 111/300 [03:10<05:03,  1.61s/it]
2024-08-07 19:36:26,149 - INFO: train loss: 0.39:  38%|###8      | 114/300 [03:14<04:59,  1.61s/it]
2024-08-07 19:36:31,358 - INFO: train loss: 0.33:  39%|###9      | 117/300 [03:20<05:01,  1.65s/it]
2024-08-07 19:36:36,841 - INFO: train loss: 0.51:  40%|####      | 120/300 [03:25<05:06,  1.70s/it]
2024-08-07 19:36:42,087 - INFO: train loss: 0.39:  41%|####1     | 123/300 [03:30<05:03,  1.72s/it]
2024-08-07 19:36:47,409 - INFO: train loss: 0.41:  42%|####2     | 126/300 [03:36<05:01,  1.73s/it]
2024-08-07 19:36:52,441 - INFO: train loss: 0.30:  43%|####3     | 129/300 [03:41<04:53,  1.72s/it]
2024-08-07 19:36:58,205 - INFO: train loss: 0.25:  44%|####4     | 132/300 [03:46<04:58,  1.78s/it]
2024-08-07 19:37:03,268 - INFO: train loss: 0.33:  45%|####5     | 135/300 [03:52<04:48,  1.75s/it]
2024-08-07 19:37:08,495 - INFO: train loss: 0.46:  46%|####6     | 138/300 [03:57<04:43,  1.75s/it]
2024-08-07 19:37:13,355 - INFO: train loss: 0.57:  47%|####6     | 141/300 [04:02<04:31,  1.71s/it]
2024-08-07 19:37:18,077 - INFO: train loss: 0.57:  48%|####8     | 144/300 [04:06<04:20,  1.67s/it]
2024-08-07 19:37:23,259 - INFO: train loss: 0.47:  49%|####9     | 147/300 [04:12<04:18,  1.69s/it]
2024-08-07 19:37:28,419 - INFO: train loss: 0.38:  50%|#####     | 150/300 [04:17<04:14,  1.70s/it]
2024-08-07 19:37:33,391 - INFO: train loss: 0.30:  51%|#####1    | 153/300 [04:22<04:07,  1.68s/it]
2024-08-07 19:37:38,529 - INFO: train loss: 0.18:  52%|#####2    | 156/300 [04:27<04:03,  1.69s/it]
2024-08-07 19:37:43,360 - INFO: train loss: 0.20:  53%|#####3    | 159/300 [04:32<03:55,  1.67s/it]
2024-08-07 19:37:48,457 - INFO: train loss: 0.24:  54%|#####4    | 162/300 [04:37<03:51,  1.68s/it]
2024-08-07 19:37:53,495 - INFO: train loss: 0.24:  55%|#####5    | 165/300 [04:42<03:46,  1.68s/it]
2024-08-07 19:37:58,752 - INFO: train loss: 0.30:  56%|#####6    | 168/300 [04:47<03:44,  1.70s/it]
2024-08-07 19:38:03,605 - INFO: train loss: 0.25:  57%|#####6    | 171/300 [04:52<03:36,  1.68s/it]
2024-08-07 19:38:08,535 - INFO: train loss: 0.24:  58%|#####8    | 174/300 [04:57<03:29,  1.67s/it]
2024-08-07 19:38:13,758 - INFO: train loss: 0.26:  59%|#####8    | 177/300 [05:02<03:27,  1.69s/it]
2024-08-07 19:38:18,430 - INFO: train loss: 0.23:  60%|######    | 180/300 [05:07<03:17,  1.65s/it]
2024-08-07 19:38:23,276 - INFO: train loss: 0.23:  61%|######1   | 183/300 [05:12<03:11,  1.64s/it]
2024-08-07 19:38:27,629 - INFO: train loss: 0.22:  62%|######2   | 186/300 [05:16<03:00,  1.58s/it]
2024-08-07 19:38:32,161 - INFO: train loss: 0.20:  63%|######3   | 189/300 [05:20<02:53,  1.56s/it]
2024-08-07 19:38:37,103 - INFO: train loss: 0.17:  64%|######4   | 192/300 [05:25<02:51,  1.59s/it]
2024-08-07 19:38:42,118 - INFO: train loss: 0.14:  65%|######5   | 195/300 [05:30<02:49,  1.61s/it]
2024-08-07 19:38:47,364 - INFO: train loss: 0.13:  66%|######6   | 198/300 [05:36<02:48,  1.65s/it]
2024-08-07 19:38:52,135 - INFO: train loss: 0.24:  67%|######7   | 201/300 [05:40<02:41,  1.63s/it]
2024-08-07 19:38:57,313 - INFO: train loss: 0.28:  68%|######8   | 204/300 [05:46<02:39,  1.66s/it]
2024-08-07 19:39:02,442 - INFO: train loss: 0.25:  69%|######9   | 207/300 [05:51<02:35,  1.68s/it]
2024-08-07 19:39:07,498 - INFO: train loss: 0.17:  70%|#######   | 210/300 [05:56<02:31,  1.68s/it]
2024-08-07 19:39:12,446 - INFO: train loss: 0.21:  71%|#######1  | 213/300 [06:01<02:25,  1.67s/it]
2024-08-07 19:39:17,397 - INFO: train loss: 0.23:  72%|#######2  | 216/300 [06:06<02:19,  1.66s/it]
2024-08-07 19:39:22,203 - INFO: train loss: 0.31:  73%|#######3  | 219/300 [06:10<02:13,  1.65s/it]
2024-08-07 19:39:27,278 - INFO: train loss: 0.36:  74%|#######4  | 222/300 [06:16<02:09,  1.66s/it]
2024-08-07 19:39:32,219 - INFO: train loss: 0.29:  75%|#######5  | 225/300 [06:20<02:04,  1.66s/it]
2024-08-07 19:39:37,287 - INFO: train loss: 0.38:  76%|#######6  | 228/300 [06:26<01:59,  1.67s/it]
2024-08-07 19:39:41,725 - INFO: train loss: 0.33:  77%|#######7  | 231/300 [06:30<01:51,  1.61s/it]
2024-08-07 19:39:46,445 - INFO: train loss: 0.44:  78%|#######8  | 234/300 [06:35<01:45,  1.60s/it]
2024-08-07 19:39:51,907 - INFO: train loss: 0.36:  79%|#######9  | 237/300 [06:40<01:44,  1.67s/it]
2024-08-07 19:39:56,707 - INFO: train loss: 0.34:  80%|########  | 240/300 [06:45<01:38,  1.65s/it]
2024-08-07 19:40:02,407 - INFO: train loss: 0.34:  81%|########1 | 243/300 [06:51<01:38,  1.72s/it]
2024-08-07 19:40:07,586 - INFO: train loss: 0.36:  82%|########2 | 246/300 [06:56<01:33,  1.72s/it]
2024-08-07 19:40:12,053 - INFO: train loss: 0.39:  83%|########2 | 249/300 [07:00<01:24,  1.65s/it]
2024-08-07 19:40:16,976 - INFO: train loss: 0.35:  84%|########4 | 252/300 [07:05<01:19,  1.65s/it]
2024-08-07 19:40:22,063 - INFO: train loss: 0.31:  85%|########5 | 255/300 [07:10<01:14,  1.66s/it]
2024-08-07 19:40:27,136 - INFO: train loss: 0.34:  86%|########6 | 258/300 [07:15<01:10,  1.67s/it]
2024-08-07 19:40:31,927 - INFO: train loss: 0.31:  87%|########7 | 261/300 [07:20<01:04,  1.65s/it]
2024-08-07 19:40:37,124 - INFO: train loss: 0.32:  88%|########8 | 264/300 [07:25<01:00,  1.67s/it]
2024-08-07 19:40:42,257 - INFO: train loss: 0.29:  89%|########9 | 267/300 [07:31<00:55,  1.69s/it]
2024-08-07 19:40:47,569 - INFO: train loss: 0.21:  90%|######### | 270/300 [07:36<00:51,  1.71s/it]
2024-08-07 19:40:52,816 - INFO: train loss: 0.25:  91%|#########1| 273/300 [07:41<00:46,  1.72s/it]
2024-08-07 19:40:57,781 - INFO: train loss: 0.35:  92%|#########2| 276/300 [07:46<00:40,  1.70s/it]
2024-08-07 19:41:02,562 - INFO: train loss: 0.33:  93%|#########3| 279/300 [07:51<00:35,  1.67s/it]
2024-08-07 19:41:07,783 - INFO: train loss: 0.37:  94%|#########3| 282/300 [07:56<00:30,  1.69s/it]
2024-08-07 19:41:12,864 - INFO: train loss: 0.22:  95%|#########5| 285/300 [08:01<00:25,  1.69s/it]
2024-08-07 19:41:17,798 - INFO: train loss: 0.27:  96%|#########6| 288/300 [08:06<00:20,  1.68s/it]
2024-08-07 19:41:23,540 - INFO: train loss: 0.37:  97%|#########7| 291/300 [08:12<00:15,  1.75s/it]
2024-08-07 19:41:28,276 - INFO: train loss: 0.36:  98%|#########8| 294/300 [08:17<00:10,  1.70s/it]
2024-08-07 19:41:33,224 - INFO: train loss: 0.41:  99%|#########9| 297/300 [08:21<00:05,  1.68s/it]
2024-08-07 19:41:39,216 - INFO: train loss: 0.36: 100%|##########| 300/300 [08:27<00:00,  1.78s/it]
2024-08-07 19:41:39,216 - INFO: train loss: 0.36: 100%|##########| 300/300 [08:27<00:00,  1.69s/it]
2024-08-07 19:41:39,216 - INFO: Saving last model checkpoint to /app/output
2024-08-07 19:41:39,216 - INFO: Saving checkpoint..
2024-08-07 19:42:06,037 - INFO: Starting validation inference
2024-08-07 19:42:06,038 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 19:42:11,243 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:04,  1.73s/it]
2024-08-07 19:42:12,763 - INFO: validation progress:   8%|8         | 6/75 [00:06<01:09,  1.01s/it]
2024-08-07 19:42:14,196 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:50,  1.30it/s]
2024-08-07 19:42:15,696 - INFO: validation progress:  16%|#6        | 12/75 [00:09<00:41,  1.51it/s]
2024-08-07 19:42:17,301 - INFO: validation progress:  20%|##        | 15/75 [00:11<00:36,  1.62it/s]
2024-08-07 19:42:18,857 - INFO: validation progress:  24%|##4       | 18/75 [00:12<00:33,  1.71it/s]
2024-08-07 19:42:20,493 - INFO: validation progress:  28%|##8       | 21/75 [00:14<00:30,  1.75it/s]
2024-08-07 19:42:21,990 - INFO: validation progress:  32%|###2      | 24/75 [00:15<00:27,  1.83it/s]
2024-08-07 19:42:23,631 - INFO: validation progress:  36%|###6      | 27/75 [00:17<00:26,  1.83it/s]
2024-08-07 19:42:25,210 - INFO: validation progress:  40%|####      | 30/75 [00:19<00:24,  1.85it/s]
2024-08-07 19:42:26,849 - INFO: validation progress:  44%|####4     | 33/75 [00:20<00:22,  1.84it/s]
2024-08-07 19:42:28,525 - INFO: validation progress:  48%|####8     | 36/75 [00:22<00:21,  1.83it/s]
2024-08-07 19:42:30,182 - INFO: validation progress:  52%|#####2    | 39/75 [00:24<00:19,  1.82it/s]
2024-08-07 19:42:31,759 - INFO: validation progress:  56%|#####6    | 42/75 [00:25<00:17,  1.85it/s]
2024-08-07 19:42:33,355 - INFO: validation progress:  60%|######    | 45/75 [00:27<00:16,  1.86it/s]
2024-08-07 19:42:34,826 - INFO: validation progress:  64%|######4   | 48/75 [00:28<00:14,  1.91it/s]
2024-08-07 19:42:36,330 - INFO: validation progress:  68%|######8   | 51/75 [00:30<00:12,  1.93it/s]
2024-08-07 19:42:37,907 - INFO: validation progress:  72%|#######2  | 54/75 [00:31<00:10,  1.92it/s]
2024-08-07 19:42:39,480 - INFO: validation progress:  76%|#######6  | 57/75 [00:33<00:09,  1.92it/s]
2024-08-07 19:42:40,919 - INFO: validation progress:  80%|########  | 60/75 [00:34<00:07,  1.97it/s]
2024-08-07 19:42:42,624 - INFO: validation progress:  84%|########4 | 63/75 [00:36<00:06,  1.90it/s]
2024-08-07 19:42:44,257 - INFO: validation progress:  88%|########8 | 66/75 [00:38<00:04,  1.88it/s]
2024-08-07 19:42:45,889 - INFO: validation progress:  92%|#########2| 69/75 [00:39<00:03,  1.87it/s]
2024-08-07 19:42:47,351 - INFO: validation progress:  96%|#########6| 72/75 [00:41<00:01,  1.92it/s]
2024-08-07 19:42:49,081 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.86it/s]
2024-08-07 19:42:49,083 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.74it/s]
2024-08-07 19:42:49,341 - INFO: Validation AUC: 0.97709
2024-08-07 19:42:49,342 - INFO: Mean validation loss: 0.49407
2024-08-07 19:42:49,692 - INFO: Training Epoch: 5 / 10
2024-08-07 19:42:49,693 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 19:42:53,545 - INFO: Evaluation step: 300
2024-08-07 19:42:59,059 - INFO: train loss: 0.27:   1%|1         | 3/300 [00:09<15:27,  3.12s/it]
2024-08-07 19:43:04,254 - INFO: train loss: 0.22:   2%|2         | 6/300 [00:14<11:17,  2.30s/it]
2024-08-07 19:43:09,394 - INFO: train loss: 0.24:   3%|3         | 9/300 [00:19<09:51,  2.03s/it]
2024-08-07 19:43:14,267 - INFO: train loss: 0.19:   4%|4         | 12/300 [00:24<08:59,  1.87s/it]
2024-08-07 19:43:19,273 - INFO: train loss: 0.19:   5%|5         | 15/300 [00:29<08:32,  1.80s/it]
2024-08-07 19:43:24,126 - INFO: train loss: 0.17:   6%|6         | 18/300 [00:34<08:09,  1.74s/it]
2024-08-07 19:43:28,745 - INFO: train loss: 0.19:   7%|7         | 21/300 [00:39<07:46,  1.67s/it]
2024-08-07 19:43:33,851 - INFO: train loss: 0.18:   8%|8         | 24/300 [00:44<07:44,  1.68s/it]
2024-08-07 19:43:38,913 - INFO: train loss: 0.19:   9%|9         | 27/300 [00:49<07:39,  1.68s/it]
2024-08-07 19:43:43,893 - INFO: train loss: 0.19:  10%|#         | 30/300 [00:54<07:32,  1.68s/it]
2024-08-07 19:43:48,883 - INFO: train loss: 0.12:  11%|#1        | 33/300 [00:59<07:26,  1.67s/it]
2024-08-07 19:43:53,384 - INFO: train loss: 0.13:  12%|#2        | 36/300 [01:03<07:07,  1.62s/it]
2024-08-07 19:43:58,510 - INFO: train loss: 0.15:  13%|#3        | 39/300 [01:08<07:09,  1.65s/it]
2024-08-07 19:44:03,554 - INFO: train loss: 0.16:  14%|#4        | 42/300 [01:13<07:07,  1.66s/it]
2024-08-07 19:44:08,495 - INFO: train loss: 0.17:  15%|#5        | 45/300 [01:18<07:01,  1.65s/it]
2024-08-07 19:44:13,812 - INFO: train loss: 0.24:  16%|#6        | 48/300 [01:24<07:05,  1.69s/it]
2024-08-07 19:44:18,769 - INFO: train loss: 0.29:  17%|#7        | 51/300 [01:29<06:57,  1.68s/it]
2024-08-07 19:44:25,098 - INFO: train loss: 0.25:  18%|#8        | 54/300 [01:35<07:24,  1.81s/it]
2024-08-07 19:44:31,233 - INFO: train loss: 0.25:  19%|#9        | 57/300 [01:41<07:36,  1.88s/it]
2024-08-07 19:44:36,038 - INFO: train loss: 0.24:  20%|##        | 60/300 [01:46<07:11,  1.80s/it]
2024-08-07 19:44:41,114 - INFO: train loss: 0.26:  21%|##1       | 63/300 [01:51<06:58,  1.76s/it]
2024-08-07 19:44:46,314 - INFO: train loss: 0.34:  22%|##2       | 66/300 [01:56<06:50,  1.76s/it]
2024-08-07 19:44:51,077 - INFO: train loss: 0.17:  23%|##3       | 69/300 [02:01<06:33,  1.70s/it]
2024-08-07 19:44:56,250 - INFO: train loss: 0.26:  24%|##4       | 72/300 [02:06<06:30,  1.71s/it]
2024-08-07 19:45:01,152 - INFO: train loss: 0.24:  25%|##5       | 75/300 [02:11<06:19,  1.69s/it]
2024-08-07 19:45:06,159 - INFO: train loss: 0.20:  26%|##6       | 78/300 [02:16<06:13,  1.68s/it]
2024-08-07 19:45:11,262 - INFO: train loss: 0.18:  27%|##7       | 81/300 [02:21<06:09,  1.69s/it]
2024-08-07 19:45:16,237 - INFO: train loss: 0.27:  28%|##8       | 84/300 [02:26<06:02,  1.68s/it]
2024-08-07 19:45:21,794 - INFO: train loss: 0.25:  29%|##9       | 87/300 [02:32<06:08,  1.73s/it]
2024-08-07 19:45:26,743 - INFO: train loss: 0.19:  30%|###       | 90/300 [02:37<05:58,  1.71s/it]
2024-08-07 19:45:31,472 - INFO: train loss: 0.20:  31%|###1      | 93/300 [02:41<05:45,  1.67s/it]
2024-08-07 19:45:36,431 - INFO: train loss: 0.20:  32%|###2      | 96/300 [02:46<05:39,  1.66s/it]
2024-08-07 19:45:41,053 - INFO: train loss: 0.23:  33%|###3      | 99/300 [02:51<05:26,  1.63s/it]
2024-08-07 19:45:46,010 - INFO: train loss: 0.22:  34%|###4      | 102/300 [02:56<05:23,  1.63s/it]
2024-08-07 19:45:50,795 - INFO: train loss: 0.19:  35%|###5      | 105/300 [03:01<05:16,  1.62s/it]
2024-08-07 19:45:55,775 - INFO: train loss: 0.21:  36%|###6      | 108/300 [03:06<05:13,  1.63s/it]
2024-08-07 19:46:00,725 - INFO: train loss: 0.23:  37%|###7      | 111/300 [03:11<05:09,  1.64s/it]
2024-08-07 19:46:05,596 - INFO: train loss: 0.19:  38%|###8      | 114/300 [03:15<05:03,  1.63s/it]
2024-08-07 19:46:11,054 - INFO: train loss: 0.16:  39%|###9      | 117/300 [03:21<05:09,  1.69s/it]
2024-08-07 19:46:15,886 - INFO: train loss: 0.21:  40%|####      | 120/300 [03:26<04:59,  1.67s/it]
2024-08-07 19:46:20,979 - INFO: train loss: 0.24:  41%|####1     | 123/300 [03:31<04:56,  1.68s/it]
2024-08-07 19:46:25,838 - INFO: train loss: 0.30:  42%|####2     | 126/300 [03:36<04:48,  1.66s/it]
2024-08-07 19:46:31,170 - INFO: train loss: 0.26:  43%|####3     | 129/300 [03:41<04:49,  1.69s/it]
2024-08-07 19:46:36,361 - INFO: train loss: 0.23:  44%|####4     | 132/300 [03:46<04:46,  1.71s/it]
2024-08-07 19:46:40,804 - INFO: train loss: 0.18:  45%|####5     | 135/300 [03:51<04:30,  1.64s/it]
2024-08-07 19:46:45,981 - INFO: train loss: 0.23:  46%|####6     | 138/300 [03:56<04:29,  1.66s/it]
2024-08-07 19:46:50,791 - INFO: train loss: 0.20:  47%|####6     | 141/300 [04:01<04:21,  1.65s/it]
2024-08-07 19:46:55,915 - INFO: train loss: 0.25:  48%|####8     | 144/300 [04:06<04:19,  1.66s/it]
2024-08-07 19:47:00,823 - INFO: train loss: 0.36:  49%|####9     | 147/300 [04:11<04:13,  1.66s/it]
2024-08-07 19:47:06,227 - INFO: train loss: 0.33:  50%|#####     | 150/300 [04:16<04:14,  1.70s/it]
2024-08-07 19:47:10,822 - INFO: train loss: 0.33:  51%|#####1    | 153/300 [04:21<04:02,  1.65s/it]
2024-08-07 19:47:15,565 - INFO: train loss: 0.28:  52%|#####2    | 156/300 [04:25<03:54,  1.63s/it]
2024-08-07 19:47:20,505 - INFO: train loss: 0.27:  53%|#####3    | 159/300 [04:30<03:50,  1.63s/it]
2024-08-07 19:47:25,483 - INFO: train loss: 0.26:  54%|#####4    | 162/300 [04:35<03:46,  1.64s/it]
2024-08-07 19:47:30,516 - INFO: train loss: 0.29:  55%|#####5    | 165/300 [04:40<03:43,  1.65s/it]
2024-08-07 19:47:35,611 - INFO: train loss: 0.32:  56%|#####6    | 168/300 [04:45<03:39,  1.67s/it]
2024-08-07 19:47:40,979 - INFO: train loss: 0.30:  57%|#####6    | 171/300 [04:51<03:39,  1.70s/it]
2024-08-07 19:47:45,953 - INFO: train loss: 0.35:  58%|#####8    | 174/300 [04:56<03:32,  1.69s/it]
2024-08-07 19:47:50,964 - INFO: train loss: 0.28:  59%|#####8    | 177/300 [05:01<03:27,  1.68s/it]
2024-08-07 19:47:56,046 - INFO: train loss: 0.17:  60%|######    | 180/300 [05:06<03:22,  1.69s/it]
2024-08-07 19:48:01,586 - INFO: train loss: 0.15:  61%|######1   | 183/300 [05:11<03:22,  1.73s/it]
2024-08-07 19:48:06,425 - INFO: train loss: 0.12:  62%|######2   | 186/300 [05:16<03:13,  1.70s/it]
2024-08-07 19:48:11,543 - INFO: train loss: 0.14:  63%|######3   | 189/300 [05:21<03:08,  1.70s/it]
2024-08-07 19:48:16,500 - INFO: train loss: 0.13:  64%|######4   | 192/300 [05:26<03:02,  1.69s/it]
2024-08-07 19:48:22,043 - INFO: train loss: 0.20:  65%|######5   | 195/300 [05:32<03:02,  1.73s/it]
2024-08-07 19:48:27,388 - INFO: train loss: 0.14:  66%|######6   | 198/300 [05:37<02:58,  1.75s/it]
2024-08-07 19:48:32,725 - INFO: train loss: 0.21:  67%|######7   | 201/300 [05:43<02:54,  1.76s/it]
2024-08-07 19:48:37,695 - INFO: train loss: 0.22:  68%|######8   | 204/300 [05:48<02:45,  1.73s/it]
2024-08-07 19:48:42,860 - INFO: train loss: 0.25:  69%|######9   | 207/300 [05:53<02:40,  1.73s/it]
2024-08-07 19:48:48,009 - INFO: train loss: 0.20:  70%|#######   | 210/300 [05:58<02:35,  1.72s/it]
2024-08-07 19:48:53,005 - INFO: train loss: 0.21:  71%|#######1  | 213/300 [06:03<02:28,  1.71s/it]
2024-08-07 19:48:57,718 - INFO: train loss: 0.21:  72%|#######2  | 216/300 [06:08<02:19,  1.67s/it]
2024-08-07 19:49:02,947 - INFO: train loss: 0.23:  73%|#######3  | 219/300 [06:13<02:16,  1.69s/it]
2024-08-07 19:49:09,310 - INFO: train loss: 0.16:  74%|#######4  | 222/300 [06:19<02:21,  1.82s/it]
2024-08-07 19:49:14,469 - INFO: train loss: 0.19:  75%|#######5  | 225/300 [06:24<02:14,  1.79s/it]
2024-08-07 19:49:19,617 - INFO: train loss: 0.16:  76%|#######6  | 228/300 [06:29<02:07,  1.77s/it]
2024-08-07 19:49:24,787 - INFO: train loss: 0.17:  77%|#######7  | 231/300 [06:35<02:01,  1.75s/it]
2024-08-07 19:49:30,512 - INFO: train loss: 0.23:  78%|#######8  | 234/300 [06:40<01:58,  1.80s/it]
2024-08-07 19:49:35,720 - INFO: train loss: 0.26:  79%|#######9  | 237/300 [06:46<01:52,  1.78s/it]
2024-08-07 19:49:40,492 - INFO: train loss: 0.31:  80%|########  | 240/300 [06:50<01:43,  1.72s/it]
2024-08-07 19:49:45,402 - INFO: train loss: 0.27:  81%|########1 | 243/300 [06:55<01:36,  1.70s/it]
2024-08-07 19:49:50,438 - INFO: train loss: 0.25:  82%|########2 | 246/300 [07:00<01:31,  1.69s/it]
2024-08-07 19:49:55,774 - INFO: train loss: 0.27:  83%|########2 | 249/300 [07:06<01:27,  1.72s/it]
2024-08-07 19:50:00,819 - INFO: train loss: 0.34:  84%|########4 | 252/300 [07:11<01:21,  1.71s/it]
2024-08-07 19:50:05,794 - INFO: train loss: 0.35:  85%|########5 | 255/300 [07:16<01:16,  1.69s/it]
2024-08-07 19:50:10,782 - INFO: train loss: 0.33:  86%|########6 | 258/300 [07:21<01:10,  1.68s/it]
2024-08-07 19:50:15,516 - INFO: train loss: 0.23:  87%|########7 | 261/300 [07:25<01:04,  1.65s/it]
2024-08-07 19:50:20,696 - INFO: train loss: 0.15:  88%|########8 | 264/300 [07:31<01:00,  1.67s/it]
2024-08-07 19:50:25,191 - INFO: train loss: 0.17:  89%|########9 | 267/300 [07:35<00:53,  1.62s/it]
2024-08-07 19:50:30,131 - INFO: train loss: 0.12:  90%|######### | 270/300 [07:40<00:48,  1.63s/it]
2024-08-07 19:50:34,990 - INFO: train loss: 0.13:  91%|#########1| 273/300 [07:45<00:43,  1.63s/it]
2024-08-07 19:50:40,272 - INFO: train loss: 0.14:  92%|#########2| 276/300 [07:50<00:39,  1.67s/it]
2024-08-07 19:50:45,388 - INFO: train loss: 0.13:  93%|#########3| 279/300 [07:55<00:35,  1.68s/it]
2024-08-07 19:50:50,418 - INFO: train loss: 0.15:  94%|#########3| 282/300 [08:00<00:30,  1.68s/it]
2024-08-07 19:50:55,431 - INFO: train loss: 0.18:  95%|#########5| 285/300 [08:05<00:25,  1.68s/it]
2024-08-07 19:51:00,418 - INFO: train loss: 0.12:  96%|#########6| 288/300 [08:10<00:20,  1.67s/it]
2024-08-07 19:51:05,401 - INFO: train loss: 0.12:  97%|#########7| 291/300 [08:15<00:15,  1.67s/it]
2024-08-07 19:51:10,634 - INFO: train loss: 0.11:  98%|#########8| 294/300 [08:20<00:10,  1.69s/it]
2024-08-07 19:51:15,379 - INFO: train loss: 0.10:  99%|#########9| 297/300 [08:25<00:04,  1.66s/it]
2024-08-07 19:51:19,991 - INFO: train loss: 0.13: 100%|##########| 300/300 [08:30<00:00,  1.62s/it]
2024-08-07 19:51:19,991 - INFO: train loss: 0.13: 100%|##########| 300/300 [08:30<00:00,  1.70s/it]
2024-08-07 19:51:19,991 - INFO: Saving last model checkpoint to /app/output
2024-08-07 19:51:19,992 - INFO: Saving checkpoint..
2024-08-07 19:51:48,053 - INFO: Starting validation inference
2024-08-07 19:51:48,053 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 19:51:53,417 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:08,  1.79s/it]
2024-08-07 19:51:54,934 - INFO: validation progress:   8%|8         | 6/75 [00:06<01:11,  1.03s/it]
2024-08-07 19:51:56,369 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:51,  1.28it/s]
2024-08-07 19:51:57,877 - INFO: validation progress:  16%|#6        | 12/75 [00:09<00:42,  1.49it/s]
2024-08-07 19:51:59,531 - INFO: validation progress:  20%|##        | 15/75 [00:11<00:37,  1.59it/s]
2024-08-07 19:52:01,132 - INFO: validation progress:  24%|##4       | 18/75 [00:13<00:33,  1.68it/s]
2024-08-07 19:52:02,775 - INFO: validation progress:  28%|##8       | 21/75 [00:14<00:31,  1.72it/s]
2024-08-07 19:52:04,267 - INFO: validation progress:  32%|###2      | 24/75 [00:16<00:28,  1.81it/s]
2024-08-07 19:52:05,909 - INFO: validation progress:  36%|###6      | 27/75 [00:17<00:26,  1.81it/s]
2024-08-07 19:52:07,490 - INFO: validation progress:  40%|####      | 30/75 [00:19<00:24,  1.84it/s]
2024-08-07 19:52:09,121 - INFO: validation progress:  44%|####4     | 33/75 [00:21<00:22,  1.84it/s]
2024-08-07 19:52:10,762 - INFO: validation progress:  48%|####8     | 36/75 [00:22<00:21,  1.84it/s]
2024-08-07 19:52:12,423 - INFO: validation progress:  52%|#####2    | 39/75 [00:24<00:19,  1.83it/s]
2024-08-07 19:52:14,011 - INFO: validation progress:  56%|#####6    | 42/75 [00:25<00:17,  1.85it/s]
2024-08-07 19:52:15,600 - INFO: validation progress:  60%|######    | 45/75 [00:27<00:16,  1.86it/s]
2024-08-07 19:52:17,059 - INFO: validation progress:  64%|######4   | 48/75 [00:29<00:14,  1.91it/s]
2024-08-07 19:52:18,565 - INFO: validation progress:  68%|######8   | 51/75 [00:30<00:12,  1.94it/s]
2024-08-07 19:52:20,141 - INFO: validation progress:  72%|#######2  | 54/75 [00:32<00:10,  1.93it/s]
2024-08-07 19:52:21,758 - INFO: validation progress:  76%|#######6  | 57/75 [00:33<00:09,  1.90it/s]
2024-08-07 19:52:23,211 - INFO: validation progress:  80%|########  | 60/75 [00:35<00:07,  1.95it/s]
2024-08-07 19:52:24,919 - INFO: validation progress:  84%|########4 | 63/75 [00:36<00:06,  1.89it/s]
2024-08-07 19:52:26,550 - INFO: validation progress:  88%|########8 | 66/75 [00:38<00:04,  1.87it/s]
2024-08-07 19:52:28,181 - INFO: validation progress:  92%|#########2| 69/75 [00:40<00:03,  1.86it/s]
2024-08-07 19:52:29,644 - INFO: validation progress:  96%|#########6| 72/75 [00:41<00:01,  1.92it/s]
2024-08-07 19:52:31,374 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.86it/s]
2024-08-07 19:52:31,378 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.73it/s]
2024-08-07 19:52:31,713 - INFO: Validation AUC: 0.98151
2024-08-07 19:52:31,713 - INFO: Mean validation loss: 0.52399
2024-08-07 19:52:32,061 - INFO: Training Epoch: 6 / 10
2024-08-07 19:52:32,062 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 19:52:35,503 - INFO: Evaluation step: 300
2024-08-07 19:52:40,151 - INFO: train loss: 0.06:   1%|1         | 3/300 [00:08<13:20,  2.70s/it]
2024-08-07 19:52:44,848 - INFO: train loss: 0.10:   2%|2         | 6/300 [00:12<09:57,  2.03s/it]
2024-08-07 19:52:50,109 - INFO: train loss: 0.09:   3%|3         | 9/300 [00:18<09:14,  1.90s/it]
2024-08-07 19:52:55,242 - INFO: train loss: 0.11:   4%|4         | 12/300 [00:23<08:46,  1.83s/it]
2024-08-07 19:53:00,339 - INFO: train loss: 0.15:   5%|5         | 15/300 [00:28<08:27,  1.78s/it]
2024-08-07 19:53:05,567 - INFO: train loss: 0.13:   6%|6         | 18/300 [00:33<08:18,  1.77s/it]
2024-08-07 19:53:10,909 - INFO: train loss: 0.11:   7%|7         | 21/300 [00:38<08:14,  1.77s/it]
2024-08-07 19:53:15,755 - INFO: train loss: 0.09:   8%|8         | 24/300 [00:43<07:55,  1.72s/it]
2024-08-07 19:53:22,147 - INFO: train loss: 0.10:   9%|9         | 27/300 [00:50<08:25,  1.85s/it]
2024-08-07 19:53:27,108 - INFO: train loss: 0.10:  10%|#         | 30/300 [00:55<08:03,  1.79s/it]
2024-08-07 19:53:32,237 - INFO: train loss: 0.08:  11%|#1        | 33/300 [01:00<07:51,  1.77s/it]
2024-08-07 19:53:36,828 - INFO: train loss: 0.08:  12%|#2        | 36/300 [01:04<07:27,  1.69s/it]
2024-08-07 19:53:41,548 - INFO: train loss: 0.05:  13%|#3        | 39/300 [01:09<07:12,  1.66s/it]
2024-08-07 19:53:46,136 - INFO: train loss: 0.08:  14%|#4        | 42/300 [01:14<06:57,  1.62s/it]
2024-08-07 19:53:51,389 - INFO: train loss: 0.14:  15%|#5        | 45/300 [01:19<07:02,  1.66s/it]
2024-08-07 19:53:56,349 - INFO: train loss: 0.14:  16%|#6        | 48/300 [01:24<06:57,  1.66s/it]
2024-08-07 19:54:01,609 - INFO: train loss: 0.16:  17%|#7        | 51/300 [01:29<06:59,  1.69s/it]
2024-08-07 19:54:06,547 - INFO: train loss: 0.15:  18%|#8        | 54/300 [01:34<06:51,  1.67s/it]
2024-08-07 19:54:11,629 - INFO: train loss: 0.12:  19%|#9        | 57/300 [01:39<06:48,  1.68s/it]
2024-08-07 19:54:16,770 - INFO: train loss: 0.12:  20%|##        | 60/300 [01:44<06:45,  1.69s/it]
2024-08-07 19:54:21,544 - INFO: train loss: 0.16:  21%|##1       | 63/300 [01:49<06:33,  1.66s/it]
2024-08-07 19:54:26,496 - INFO: train loss: 0.22:  22%|##2       | 66/300 [01:54<06:27,  1.66s/it]
2024-08-07 19:54:31,316 - INFO: train loss: 0.25:  23%|##3       | 69/300 [01:59<06:19,  1.64s/it]
2024-08-07 19:54:36,453 - INFO: train loss: 0.28:  24%|##4       | 72/300 [02:04<06:19,  1.66s/it]
2024-08-07 19:54:41,257 - INFO: train loss: 0.15:  25%|##5       | 75/300 [02:09<06:10,  1.64s/it]
2024-08-07 19:54:46,235 - INFO: train loss: 0.10:  26%|##6       | 78/300 [02:14<06:06,  1.65s/it]
2024-08-07 19:54:51,120 - INFO: train loss: 0.09:  27%|##7       | 81/300 [02:19<05:59,  1.64s/it]
2024-08-07 19:54:56,442 - INFO: train loss: 0.08:  28%|##8       | 84/300 [02:24<06:03,  1.68s/it]
2024-08-07 19:55:01,897 - INFO: train loss: 0.12:  29%|##9       | 87/300 [02:29<06:07,  1.72s/it]
2024-08-07 19:55:06,884 - INFO: train loss: 0.13:  30%|###       | 90/300 [02:34<05:58,  1.70s/it]
2024-08-07 19:55:11,905 - INFO: train loss: 0.16:  31%|###1      | 93/300 [02:39<05:50,  1.70s/it]
2024-08-07 19:55:16,948 - INFO: train loss: 0.20:  32%|###2      | 96/300 [02:44<05:44,  1.69s/it]
2024-08-07 19:55:21,933 - INFO: train loss: 0.16:  33%|###3      | 99/300 [02:49<05:38,  1.68s/it]
2024-08-07 19:55:26,751 - INFO: train loss: 0.24:  34%|###4      | 102/300 [02:54<05:28,  1.66s/it]
2024-08-07 19:55:31,702 - INFO: train loss: 0.16:  35%|###5      | 105/300 [02:59<05:23,  1.66s/it]
2024-08-07 19:55:36,914 - INFO: train loss: 0.15:  36%|###6      | 108/300 [03:04<05:22,  1.68s/it]
2024-08-07 19:55:42,618 - INFO: train loss: 0.05:  37%|###7      | 111/300 [03:10<05:30,  1.75s/it]
2024-08-07 19:55:47,962 - INFO: train loss: 0.08:  38%|###8      | 114/300 [03:15<05:26,  1.76s/it]
2024-08-07 19:55:52,860 - INFO: train loss: 0.08:  39%|###9      | 117/300 [03:20<05:14,  1.72s/it]
2024-08-07 19:55:57,510 - INFO: train loss: 0.09:  40%|####      | 120/300 [03:25<05:00,  1.67s/it]
2024-08-07 19:56:02,668 - INFO: train loss: 0.09:  41%|####1     | 123/300 [03:30<04:58,  1.68s/it]
2024-08-07 19:56:07,648 - INFO: train loss: 0.06:  42%|####2     | 126/300 [03:35<04:51,  1.68s/it]
2024-08-07 19:56:12,724 - INFO: train loss: 0.08:  43%|####3     | 129/300 [03:40<04:47,  1.68s/it]
2024-08-07 19:56:17,586 - INFO: train loss: 0.05:  44%|####4     | 132/300 [03:45<04:39,  1.66s/it]
2024-08-07 19:56:23,014 - INFO: train loss: 0.12:  45%|####5     | 135/300 [03:50<04:41,  1.71s/it]
2024-08-07 19:56:27,863 - INFO: train loss: 0.16:  46%|####6     | 138/300 [03:55<04:32,  1.68s/it]
2024-08-07 19:56:32,610 - INFO: train loss: 0.14:  47%|####6     | 141/300 [04:00<04:22,  1.65s/it]
2024-08-07 19:56:38,024 - INFO: train loss: 0.11:  48%|####8     | 144/300 [04:05<04:24,  1.70s/it]
2024-08-07 19:56:42,919 - INFO: train loss: 0.12:  49%|####9     | 147/300 [04:10<04:16,  1.68s/it]
2024-08-07 19:56:47,811 - INFO: train loss: 0.12:  50%|#####     | 150/300 [04:15<04:09,  1.66s/it]
2024-08-07 19:56:52,735 - INFO: train loss: 0.12:  51%|#####1    | 153/300 [04:20<04:03,  1.66s/it]
2024-08-07 19:56:58,101 - INFO: train loss: 0.13:  52%|#####2    | 156/300 [04:26<04:04,  1.70s/it]
2024-08-07 19:57:03,301 - INFO: train loss: 0.04:  53%|#####3    | 159/300 [04:31<04:00,  1.71s/it]
2024-08-07 19:57:08,129 - INFO: train loss: 0.06:  54%|#####4    | 162/300 [04:36<03:51,  1.68s/it]
2024-08-07 19:57:12,945 - INFO: train loss: 0.09:  55%|#####5    | 165/300 [04:40<03:43,  1.66s/it]
2024-08-07 19:57:18,450 - INFO: train loss: 0.06:  56%|#####6    | 168/300 [04:46<03:45,  1.71s/it]
2024-08-07 19:57:23,372 - INFO: train loss: 0.06:  57%|#####6    | 171/300 [04:51<03:37,  1.69s/it]
2024-08-07 19:57:28,187 - INFO: train loss: 0.08:  58%|#####8    | 174/300 [04:56<03:29,  1.66s/it]
2024-08-07 19:57:33,906 - INFO: train loss: 0.06:  59%|#####8    | 177/300 [05:01<03:33,  1.74s/it]
2024-08-07 19:57:38,763 - INFO: train loss: 0.08:  60%|######    | 180/300 [05:06<03:24,  1.70s/it]
2024-08-07 19:57:44,154 - INFO: train loss: 0.16:  61%|######1   | 183/300 [05:12<03:22,  1.73s/it]
2024-08-07 19:57:49,263 - INFO: train loss: 0.17:  62%|######2   | 186/300 [05:17<03:16,  1.72s/it]
2024-08-07 19:57:54,682 - INFO: train loss: 0.18:  63%|######3   | 189/300 [05:22<03:13,  1.75s/it]
2024-08-07 19:57:59,696 - INFO: train loss: 0.08:  64%|######4   | 192/300 [05:27<03:06,  1.72s/it]
2024-08-07 19:58:04,539 - INFO: train loss: 0.09:  65%|######5   | 195/300 [05:32<02:57,  1.69s/it]
2024-08-07 19:58:09,574 - INFO: train loss: 0.07:  66%|######6   | 198/300 [05:37<02:52,  1.69s/it]
2024-08-07 19:58:14,399 - INFO: train loss: 0.08:  67%|######7   | 201/300 [05:42<02:44,  1.66s/it]
2024-08-07 19:58:19,098 - INFO: train loss: 0.05:  68%|######8   | 204/300 [05:47<02:36,  1.63s/it]
2024-08-07 19:58:23,987 - INFO: train loss: 0.04:  69%|######9   | 207/300 [05:51<02:31,  1.63s/it]
2024-08-07 19:58:28,915 - INFO: train loss: 0.07:  70%|#######   | 210/300 [05:56<02:27,  1.64s/it]
2024-08-07 19:58:34,033 - INFO: train loss: 0.09:  71%|#######1  | 213/300 [06:01<02:24,  1.66s/it]
2024-08-07 19:58:38,472 - INFO: train loss: 0.09:  72%|#######2  | 216/300 [06:06<02:14,  1.60s/it]
2024-08-07 19:58:43,909 - INFO: train loss: 0.15:  73%|#######3  | 219/300 [06:11<02:14,  1.67s/it]
2024-08-07 19:58:48,658 - INFO: train loss: 0.11:  74%|#######4  | 222/300 [06:16<02:08,  1.64s/it]
2024-08-07 19:58:53,939 - INFO: train loss: 0.11:  75%|#######5  | 225/300 [06:21<02:05,  1.68s/it]
2024-08-07 19:58:58,561 - INFO: train loss: 0.09:  76%|#######6  | 228/300 [06:26<01:57,  1.64s/it]
2024-08-07 19:59:04,000 - INFO: train loss: 0.05:  77%|#######7  | 231/300 [06:31<01:56,  1.69s/it]
2024-08-07 19:59:08,977 - INFO: train loss: 0.09:  78%|#######8  | 234/300 [06:36<01:50,  1.68s/it]
2024-08-07 19:59:14,056 - INFO: train loss: 0.07:  79%|#######9  | 237/300 [06:41<01:46,  1.68s/it]
2024-08-07 19:59:19,292 - INFO: train loss: 0.09:  80%|########  | 240/300 [06:47<01:42,  1.70s/it]
2024-08-07 19:59:24,034 - INFO: train loss: 0.06:  81%|########1 | 243/300 [06:51<01:34,  1.67s/it]
2024-08-07 19:59:28,841 - INFO: train loss: 0.06:  82%|########2 | 246/300 [06:56<01:28,  1.65s/it]
2024-08-07 19:59:33,475 - INFO: train loss: 0.09:  83%|########2 | 249/300 [07:01<01:22,  1.62s/it]
2024-08-07 19:59:38,769 - INFO: train loss: 0.14:  84%|########4 | 252/300 [07:06<01:19,  1.66s/it]
2024-08-07 19:59:45,661 - INFO: train loss: 0.13:  85%|########5 | 255/300 [07:13<01:23,  1.85s/it]
2024-08-07 19:59:50,852 - INFO: train loss: 0.12:  86%|########6 | 258/300 [07:18<01:16,  1.82s/it]
2024-08-07 19:59:55,773 - INFO: train loss: 0.10:  87%|########7 | 261/300 [07:23<01:08,  1.76s/it]
2024-08-07 20:00:00,844 - INFO: train loss: 0.06:  88%|########8 | 264/300 [07:28<01:02,  1.74s/it]
2024-08-07 20:00:05,670 - INFO: train loss: 0.06:  89%|########9 | 267/300 [07:33<00:56,  1.70s/it]
2024-08-07 20:00:10,560 - INFO: train loss: 0.06:  90%|######### | 270/300 [07:38<00:50,  1.68s/it]
2024-08-07 20:00:15,214 - INFO: train loss: 0.04:  91%|#########1| 273/300 [07:43<00:44,  1.64s/it]
2024-08-07 20:00:20,459 - INFO: train loss: 0.05:  92%|#########2| 276/300 [07:48<00:40,  1.67s/it]
2024-08-07 20:00:25,706 - INFO: train loss: 0.04:  93%|#########3| 279/300 [07:53<00:35,  1.70s/it]
2024-08-07 20:00:30,506 - INFO: train loss: 0.04:  94%|#########3| 282/300 [07:58<00:30,  1.67s/it]
2024-08-07 20:00:35,468 - INFO: train loss: 0.04:  95%|#########5| 285/300 [08:03<00:24,  1.66s/it]
2024-08-07 20:00:40,906 - INFO: train loss: 0.07:  96%|#########6| 288/300 [08:08<00:20,  1.71s/it]
2024-08-07 20:00:46,065 - INFO: train loss: 0.09:  97%|#########7| 291/300 [08:14<00:15,  1.71s/it]
2024-08-07 20:00:51,141 - INFO: train loss: 0.09:  98%|#########8| 294/300 [08:19<00:10,  1.71s/it]
2024-08-07 20:00:56,616 - INFO: train loss: 0.08:  99%|#########9| 297/300 [08:24<00:05,  1.74s/it]
2024-08-07 20:01:01,959 - INFO: train loss: 0.06: 100%|##########| 300/300 [08:29<00:00,  1.75s/it]
2024-08-07 20:01:01,959 - INFO: train loss: 0.06: 100%|##########| 300/300 [08:29<00:00,  1.70s/it]
2024-08-07 20:01:01,959 - INFO: Saving last model checkpoint to /app/output
2024-08-07 20:01:01,960 - INFO: Saving checkpoint..
2024-08-07 20:01:27,878 - INFO: Starting validation inference
2024-08-07 20:01:27,879 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 20:01:33,273 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:09,  1.80s/it]
2024-08-07 20:01:34,791 - INFO: validation progress:   8%|8         | 6/75 [00:06<01:11,  1.04s/it]
2024-08-07 20:01:36,225 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:51,  1.28it/s]
2024-08-07 20:01:37,727 - INFO: validation progress:  16%|#6        | 12/75 [00:09<00:42,  1.49it/s]
2024-08-07 20:01:39,331 - INFO: validation progress:  20%|##        | 15/75 [00:11<00:37,  1.61it/s]
2024-08-07 20:01:40,888 - INFO: validation progress:  24%|##4       | 18/75 [00:13<00:33,  1.70it/s]
2024-08-07 20:01:42,520 - INFO: validation progress:  28%|##8       | 21/75 [00:14<00:30,  1.75it/s]
2024-08-07 20:01:44,012 - INFO: validation progress:  32%|###2      | 24/75 [00:16<00:27,  1.82it/s]
2024-08-07 20:01:45,653 - INFO: validation progress:  36%|###6      | 27/75 [00:17<00:26,  1.82it/s]
2024-08-07 20:01:47,236 - INFO: validation progress:  40%|####      | 30/75 [00:19<00:24,  1.85it/s]
2024-08-07 20:01:48,871 - INFO: validation progress:  44%|####4     | 33/75 [00:20<00:22,  1.84it/s]
2024-08-07 20:01:50,510 - INFO: validation progress:  48%|####8     | 36/75 [00:22<00:21,  1.84it/s]
2024-08-07 20:01:52,169 - INFO: validation progress:  52%|#####2    | 39/75 [00:24<00:19,  1.83it/s]
2024-08-07 20:01:53,742 - INFO: validation progress:  56%|#####6    | 42/75 [00:25<00:17,  1.85it/s]
2024-08-07 20:01:55,302 - INFO: validation progress:  60%|######    | 45/75 [00:27<00:16,  1.87it/s]
2024-08-07 20:01:56,798 - INFO: validation progress:  64%|######4   | 48/75 [00:28<00:14,  1.91it/s]
2024-08-07 20:01:58,300 - INFO: validation progress:  68%|######8   | 51/75 [00:30<00:12,  1.94it/s]
2024-08-07 20:01:59,925 - INFO: validation progress:  72%|#######2  | 54/75 [00:32<00:11,  1.91it/s]
2024-08-07 20:02:01,517 - INFO: validation progress:  76%|#######6  | 57/75 [00:33<00:09,  1.90it/s]
2024-08-07 20:02:02,957 - INFO: validation progress:  80%|########  | 60/75 [00:35<00:07,  1.95it/s]
2024-08-07 20:02:04,662 - INFO: validation progress:  84%|########4 | 63/75 [00:36<00:06,  1.89it/s]
2024-08-07 20:02:06,292 - INFO: validation progress:  88%|########8 | 66/75 [00:38<00:04,  1.88it/s]
2024-08-07 20:02:07,923 - INFO: validation progress:  92%|#########2| 69/75 [00:40<00:03,  1.86it/s]
2024-08-07 20:02:09,386 - INFO: validation progress:  96%|#########6| 72/75 [00:41<00:01,  1.92it/s]
2024-08-07 20:02:11,117 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.86it/s]
2024-08-07 20:02:11,121 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.73it/s]
2024-08-07 20:02:11,406 - INFO: Validation AUC: 0.98634
2024-08-07 20:02:11,406 - INFO: Mean validation loss: 0.59212
2024-08-07 20:02:11,743 - INFO: Training Epoch: 7 / 10
2024-08-07 20:02:11,743 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 20:02:15,215 - INFO: Evaluation step: 300
2024-08-07 20:02:20,317 - INFO: train loss: 0.03:   1%|1         | 3/300 [00:08<14:08,  2.86s/it]
2024-08-07 20:02:25,440 - INFO: train loss: 0.05:   2%|2         | 6/300 [00:13<10:41,  2.18s/it]
2024-08-07 20:02:30,534 - INFO: train loss: 0.03:   3%|3         | 9/300 [00:18<09:30,  1.96s/it]
2024-08-07 20:02:35,772 - INFO: train loss: 0.02:   4%|4         | 12/300 [00:24<09:00,  1.88s/it]
2024-08-07 20:02:41,171 - INFO: train loss: 0.03:   5%|5         | 15/300 [00:29<08:46,  1.85s/it]
2024-08-07 20:02:46,361 - INFO: train loss: 0.01:   6%|6         | 18/300 [00:34<08:29,  1.81s/it]
2024-08-07 20:02:51,986 - INFO: train loss: 0.02:   7%|7         | 21/300 [00:40<08:30,  1.83s/it]
2024-08-07 20:02:56,682 - INFO: train loss: 0.03:   8%|8         | 24/300 [00:44<08:01,  1.75s/it]
2024-08-07 20:03:01,644 - INFO: train loss: 0.03:   9%|9         | 27/300 [00:49<07:48,  1.72s/it]
2024-08-07 20:03:07,004 - INFO: train loss: 0.03:  10%|#         | 30/300 [00:55<07:49,  1.74s/it]
2024-08-07 20:03:12,281 - INFO: train loss: 0.02:  11%|#1        | 33/300 [01:00<07:45,  1.74s/it]
2024-08-07 20:03:17,109 - INFO: train loss: 0.01:  12%|#2        | 36/300 [01:05<07:29,  1.70s/it]
2024-08-07 20:03:21,979 - INFO: train loss: 0.01:  13%|#3        | 39/300 [01:10<07:18,  1.68s/it]
2024-08-07 20:03:26,998 - INFO: train loss: 0.03:  14%|#4        | 42/300 [01:15<07:12,  1.68s/it]
2024-08-07 20:03:32,329 - INFO: train loss: 0.08:  15%|#5        | 45/300 [01:20<07:15,  1.71s/it]
2024-08-07 20:03:38,048 - INFO: train loss: 0.09:  16%|#6        | 48/300 [01:26<07:25,  1.77s/it]
2024-08-07 20:03:43,786 - INFO: train loss: 0.09:  17%|#7        | 51/300 [01:32<07:30,  1.81s/it]
2024-08-07 20:03:48,612 - INFO: train loss: 0.08:  18%|#8        | 54/300 [01:36<07:10,  1.75s/it]
2024-08-07 20:03:53,591 - INFO: train loss: 0.05:  19%|#9        | 57/300 [01:41<06:58,  1.72s/it]
2024-08-07 20:03:58,413 - INFO: train loss: 0.05:  20%|##        | 60/300 [01:46<06:45,  1.69s/it]
2024-08-07 20:04:03,882 - INFO: train loss: 0.04:  21%|##1       | 63/300 [01:52<06:49,  1.73s/it]
2024-08-07 20:04:09,110 - INFO: train loss: 0.08:  22%|##2       | 66/300 [01:57<06:45,  1.73s/it]
2024-08-07 20:04:14,332 - INFO: train loss: 0.06:  23%|##3       | 69/300 [02:02<06:40,  1.74s/it]
2024-08-07 20:04:19,200 - INFO: train loss: 0.06:  24%|##4       | 72/300 [02:07<06:27,  1.70s/it]
2024-08-07 20:04:24,256 - INFO: train loss: 0.03:  25%|##5       | 75/300 [02:12<06:21,  1.70s/it]
2024-08-07 20:04:29,059 - INFO: train loss: 0.02:  26%|##6       | 78/300 [02:17<06:10,  1.67s/it]
2024-08-07 20:04:33,717 - INFO: train loss: 0.08:  27%|##7       | 81/300 [02:21<05:57,  1.63s/it]
2024-08-07 20:04:38,820 - INFO: train loss: 0.09:  28%|##8       | 84/300 [02:27<05:57,  1.65s/it]
2024-08-07 20:04:43,646 - INFO: train loss: 0.09:  29%|##9       | 87/300 [02:31<05:49,  1.64s/it]
2024-08-07 20:04:48,597 - INFO: train loss: 0.05:  30%|###       | 90/300 [02:36<05:45,  1.64s/it]
2024-08-07 20:04:53,615 - INFO: train loss: 0.02:  31%|###1      | 93/300 [02:41<05:41,  1.65s/it]
2024-08-07 20:04:58,559 - INFO: train loss: 0.02:  32%|###2      | 96/300 [02:46<05:36,  1.65s/it]
2024-08-07 20:05:03,532 - INFO: train loss: 0.02:  33%|###3      | 99/300 [02:51<05:32,  1.65s/it]
2024-08-07 20:05:08,245 - INFO: train loss: 0.02:  34%|###4      | 102/300 [02:56<05:22,  1.63s/it]
2024-08-07 20:05:13,434 - INFO: train loss: 0.01:  35%|###5      | 105/300 [03:01<05:23,  1.66s/it]
2024-08-07 20:05:18,070 - INFO: train loss: 0.01:  36%|###6      | 108/300 [03:06<05:11,  1.62s/it]
2024-08-07 20:05:23,459 - INFO: train loss: 0.02:  37%|###7      | 111/300 [03:11<05:16,  1.68s/it]
2024-08-07 20:05:28,562 - INFO: train loss: 0.02:  38%|###8      | 114/300 [03:16<05:13,  1.68s/it]
2024-08-07 20:05:33,874 - INFO: train loss: 0.02:  39%|###9      | 117/300 [03:22<05:12,  1.71s/it]
2024-08-07 20:05:39,427 - INFO: train loss: 0.01:  40%|####      | 120/300 [03:27<05:15,  1.75s/it]
2024-08-07 20:05:44,155 - INFO: train loss: 0.01:  41%|####1     | 123/300 [03:32<05:00,  1.70s/it]
2024-08-07 20:05:49,161 - INFO: train loss: 0.01:  42%|####2     | 126/300 [03:37<04:54,  1.69s/it]
2024-08-07 20:05:54,090 - INFO: train loss: 0.02:  43%|####3     | 129/300 [03:42<04:46,  1.68s/it]
2024-08-07 20:05:59,132 - INFO: train loss: 0.02:  44%|####4     | 132/300 [03:47<04:41,  1.68s/it]
2024-08-07 20:06:03,890 - INFO: train loss: 0.05:  45%|####5     | 135/300 [03:52<04:32,  1.65s/it]
2024-08-07 20:06:08,624 - INFO: train loss: 0.06:  46%|####6     | 138/300 [03:56<04:23,  1.63s/it]
2024-08-07 20:06:13,985 - INFO: train loss: 0.07:  47%|####6     | 141/300 [04:02<04:26,  1.68s/it]
2024-08-07 20:06:18,920 - INFO: train loss: 0.04:  48%|####8     | 144/300 [04:07<04:20,  1.67s/it]
2024-08-07 20:06:23,726 - INFO: train loss: 0.03:  49%|####9     | 147/300 [04:11<04:12,  1.65s/it]
2024-08-07 20:06:28,750 - INFO: train loss: 0.02:  50%|#####     | 150/300 [04:17<04:08,  1.66s/it]
2024-08-07 20:06:33,461 - INFO: train loss: 0.04:  51%|#####1    | 153/300 [04:21<03:59,  1.63s/it]
2024-08-07 20:06:38,503 - INFO: train loss: 0.03:  52%|#####2    | 156/300 [04:26<03:56,  1.65s/it]
2024-08-07 20:06:43,401 - INFO: train loss: 0.03:  53%|#####3    | 159/300 [04:31<03:51,  1.64s/it]
2024-08-07 20:06:48,369 - INFO: train loss: 0.03:  54%|#####4    | 162/300 [04:36<03:47,  1.65s/it]
2024-08-07 20:06:53,627 - INFO: train loss: 0.02:  55%|#####5    | 165/300 [04:41<03:46,  1.68s/it]
2024-08-07 20:06:58,654 - INFO: train loss: 0.02:  56%|#####6    | 168/300 [04:46<03:41,  1.68s/it]
2024-08-07 20:07:04,980 - INFO: train loss: 0.03:  57%|#####6    | 171/300 [04:53<03:53,  1.81s/it]
2024-08-07 20:07:10,211 - INFO: train loss: 0.02:  58%|#####8    | 174/300 [04:58<03:45,  1.79s/it]
2024-08-07 20:07:15,070 - INFO: train loss: 0.02:  59%|#####8    | 177/300 [05:03<03:33,  1.74s/it]
2024-08-07 20:07:19,764 - INFO: train loss: 0.04:  60%|######    | 180/300 [05:08<03:22,  1.69s/it]
2024-08-07 20:07:24,692 - INFO: train loss: 0.03:  61%|######1   | 183/300 [05:12<03:15,  1.67s/it]
2024-08-07 20:07:29,838 - INFO: train loss: 0.06:  62%|######2   | 186/300 [05:18<03:12,  1.69s/it]
2024-08-07 20:07:34,578 - INFO: train loss: 0.04:  63%|######3   | 189/300 [05:22<03:03,  1.65s/it]
2024-08-07 20:07:39,734 - INFO: train loss: 0.04:  64%|######4   | 192/300 [05:27<03:00,  1.67s/it]
2024-08-07 20:07:44,805 - INFO: train loss: 0.02:  65%|######5   | 195/300 [05:33<02:56,  1.68s/it]
2024-08-07 20:07:49,962 - INFO: train loss: 0.03:  66%|######6   | 198/300 [05:38<02:52,  1.69s/it]
2024-08-07 20:07:55,180 - INFO: train loss: 0.03:  67%|######7   | 201/300 [05:43<02:48,  1.71s/it]
2024-08-07 20:08:00,138 - INFO: train loss: 0.03:  68%|######8   | 204/300 [05:48<02:42,  1.69s/it]
2024-08-07 20:08:05,303 - INFO: train loss: 0.03:  69%|######9   | 207/300 [05:53<02:38,  1.70s/it]
2024-08-07 20:08:09,990 - INFO: train loss: 0.03:  70%|#######   | 210/300 [05:58<02:29,  1.66s/it]
2024-08-07 20:08:15,237 - INFO: train loss: 0.02:  71%|#######1  | 213/300 [06:03<02:26,  1.69s/it]
2024-08-07 20:08:20,011 - INFO: train loss: 0.03:  72%|#######2  | 216/300 [06:08<02:19,  1.66s/it]
2024-08-07 20:08:25,493 - INFO: train loss: 0.01:  73%|#######3  | 219/300 [06:13<02:18,  1.71s/it]
2024-08-07 20:08:30,240 - INFO: train loss: 0.01:  74%|#######4  | 222/300 [06:18<02:10,  1.67s/it]
2024-08-07 20:08:35,411 - INFO: train loss: 0.01:  75%|#######5  | 225/300 [06:23<02:06,  1.69s/it]
2024-08-07 20:08:40,752 - INFO: train loss: 0.01:  76%|#######6  | 228/300 [06:29<02:03,  1.71s/it]
2024-08-07 20:08:45,759 - INFO: train loss: 0.02:  77%|#######7  | 231/300 [06:34<01:57,  1.70s/it]
2024-08-07 20:08:50,785 - INFO: train loss: 0.02:  78%|#######8  | 234/300 [06:39<01:51,  1.69s/it]
2024-08-07 20:08:55,851 - INFO: train loss: 0.02:  79%|#######9  | 237/300 [06:44<01:46,  1.69s/it]
2024-08-07 20:09:01,235 - INFO: train loss: 0.03:  80%|########  | 240/300 [06:49<01:43,  1.72s/it]
2024-08-07 20:09:06,816 - INFO: train loss: 0.08:  81%|########1 | 243/300 [06:55<01:40,  1.76s/it]
2024-08-07 20:09:11,993 - INFO: train loss: 0.08:  82%|########2 | 246/300 [07:00<01:34,  1.75s/it]
2024-08-07 20:09:16,995 - INFO: train loss: 0.07:  83%|########2 | 249/300 [07:05<01:28,  1.73s/it]
2024-08-07 20:09:22,471 - INFO: train loss: 0.04:  84%|########4 | 252/300 [07:10<01:24,  1.76s/it]
2024-08-07 20:09:27,640 - INFO: train loss: 0.03:  85%|########5 | 255/300 [07:15<01:18,  1.75s/it]
2024-08-07 20:09:33,109 - INFO: train loss: 0.03:  86%|########6 | 258/300 [07:21<01:14,  1.77s/it]
2024-08-07 20:09:37,615 - INFO: train loss: 0.04:  87%|########7 | 261/300 [07:25<01:05,  1.69s/it]
2024-08-07 20:09:42,610 - INFO: train loss: 0.05:  88%|########8 | 264/300 [07:30<01:00,  1.68s/it]
2024-08-07 20:09:47,847 - INFO: train loss: 0.06:  89%|########9 | 267/300 [07:36<00:56,  1.70s/it]
2024-08-07 20:09:52,913 - INFO: train loss: 0.04:  90%|######### | 270/300 [07:41<00:50,  1.70s/it]
2024-08-07 20:09:57,770 - INFO: train loss: 0.04:  91%|#########1| 273/300 [07:46<00:45,  1.67s/it]
2024-08-07 20:10:02,584 - INFO: train loss: 0.03:  92%|#########2| 276/300 [07:50<00:39,  1.65s/it]
2024-08-07 20:10:07,523 - INFO: train loss: 0.01:  93%|#########3| 279/300 [07:55<00:34,  1.65s/it]
2024-08-07 20:10:12,848 - INFO: train loss: 0.01:  94%|#########3| 282/300 [08:01<00:30,  1.69s/it]
2024-08-07 20:10:18,059 - INFO: train loss: 0.01:  95%|#########5| 285/300 [08:06<00:25,  1.70s/it]
2024-08-07 20:10:22,998 - INFO: train loss: 0.01:  96%|#########6| 288/300 [08:11<00:20,  1.69s/it]
2024-08-07 20:10:28,501 - INFO: train loss: 0.05:  97%|#########7| 291/300 [08:16<00:15,  1.73s/it]
2024-08-07 20:10:33,431 - INFO: train loss: 0.07:  98%|#########8| 294/300 [08:21<00:10,  1.70s/it]
2024-08-07 20:10:38,856 - INFO: train loss: 0.08:  99%|#########9| 297/300 [08:27<00:05,  1.74s/it]
2024-08-07 20:10:43,577 - INFO: train loss: 0.07: 100%|##########| 300/300 [08:31<00:00,  1.69s/it]
2024-08-07 20:10:43,577 - INFO: train loss: 0.07: 100%|##########| 300/300 [08:31<00:00,  1.71s/it]
2024-08-07 20:10:43,577 - INFO: Saving last model checkpoint to /app/output
2024-08-07 20:10:43,577 - INFO: Saving checkpoint..
2024-08-07 20:11:09,827 - INFO: Starting validation inference
2024-08-07 20:11:09,827 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 20:11:15,580 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:18,  1.92s/it]
2024-08-07 20:11:17,097 - INFO: validation progress:   8%|8         | 6/75 [00:07<01:15,  1.09s/it]
2024-08-07 20:11:18,540 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:53,  1.23it/s]
2024-08-07 20:11:20,041 - INFO: validation progress:  16%|#6        | 12/75 [00:10<00:43,  1.45it/s]
2024-08-07 20:11:21,646 - INFO: validation progress:  20%|##        | 15/75 [00:11<00:37,  1.58it/s]
2024-08-07 20:11:23,314 - INFO: validation progress:  24%|##4       | 18/75 [00:13<00:34,  1.65it/s]
2024-08-07 20:11:24,946 - INFO: validation progress:  28%|##8       | 21/75 [00:15<00:31,  1.71it/s]
2024-08-07 20:11:26,443 - INFO: validation progress:  32%|###2      | 24/75 [00:16<00:28,  1.79it/s]
2024-08-07 20:11:28,095 - INFO: validation progress:  36%|###6      | 27/75 [00:18<00:26,  1.80it/s]
2024-08-07 20:11:29,673 - INFO: validation progress:  40%|####      | 30/75 [00:19<00:24,  1.83it/s]
2024-08-07 20:11:31,390 - INFO: validation progress:  44%|####4     | 33/75 [00:21<00:23,  1.80it/s]
2024-08-07 20:11:33,039 - INFO: validation progress:  48%|####8     | 36/75 [00:23<00:21,  1.81it/s]
2024-08-07 20:11:34,699 - INFO: validation progress:  52%|#####2    | 39/75 [00:24<00:19,  1.81it/s]
2024-08-07 20:11:36,272 - INFO: validation progress:  56%|#####6    | 42/75 [00:26<00:17,  1.84it/s]
2024-08-07 20:11:37,827 - INFO: validation progress:  60%|######    | 45/75 [00:27<00:16,  1.86it/s]
2024-08-07 20:11:39,285 - INFO: validation progress:  64%|######4   | 48/75 [00:29<00:14,  1.92it/s]
2024-08-07 20:11:40,791 - INFO: validation progress:  68%|######8   | 51/75 [00:30<00:12,  1.94it/s]
2024-08-07 20:11:42,372 - INFO: validation progress:  72%|#######2  | 54/75 [00:32<00:10,  1.93it/s]
2024-08-07 20:11:43,953 - INFO: validation progress:  76%|#######6  | 57/75 [00:34<00:09,  1.92it/s]
2024-08-07 20:11:45,434 - INFO: validation progress:  80%|########  | 60/75 [00:35<00:07,  1.95it/s]
2024-08-07 20:11:47,188 - INFO: validation progress:  84%|########4 | 63/75 [00:37<00:06,  1.87it/s]
2024-08-07 20:11:48,833 - INFO: validation progress:  88%|########8 | 66/75 [00:39<00:04,  1.86it/s]
2024-08-07 20:11:50,465 - INFO: validation progress:  92%|#########2| 69/75 [00:40<00:03,  1.85it/s]
2024-08-07 20:11:51,930 - INFO: validation progress:  96%|#########6| 72/75 [00:42<00:01,  1.91it/s]
2024-08-07 20:11:53,661 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.85it/s]
2024-08-07 20:11:53,663 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.71it/s]
2024-08-07 20:11:53,937 - INFO: Validation AUC: 0.98353
2024-08-07 20:11:53,937 - INFO: Mean validation loss: 0.77703
2024-08-07 20:11:54,279 - INFO: Training Epoch: 8 / 10
2024-08-07 20:11:54,279 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 20:11:57,790 - INFO: Evaluation step: 300
2024-08-07 20:12:02,761 - INFO: train loss: 0.06:   1%|1         | 3/300 [00:08<13:59,  2.83s/it]
2024-08-07 20:12:07,850 - INFO: train loss: 0.03:   2%|2         | 6/300 [00:13<10:35,  2.16s/it]
2024-08-07 20:12:12,951 - INFO: train loss: 0.03:   3%|3         | 9/300 [00:18<09:27,  1.95s/it]
2024-08-07 20:12:18,228 - INFO: train loss: 0.01:   4%|4         | 12/300 [00:23<09:00,  1.88s/it]
2024-08-07 20:12:22,904 - INFO: train loss: 0.01:   5%|5         | 15/300 [00:28<08:21,  1.76s/it]
2024-08-07 20:12:27,761 - INFO: train loss: 0.01:   6%|6         | 18/300 [00:33<08:03,  1.71s/it]
2024-08-07 20:12:32,963 - INFO: train loss: 0.01:   7%|7         | 21/300 [00:38<07:59,  1.72s/it]
2024-08-07 20:12:38,069 - INFO: train loss: 0.00:   8%|8         | 24/300 [00:43<07:53,  1.71s/it]
2024-08-07 20:12:42,760 - INFO: train loss: 0.01:   9%|9         | 27/300 [00:48<07:35,  1.67s/it]
2024-08-07 20:12:47,632 - INFO: train loss: 0.02:  10%|#         | 30/300 [00:53<07:26,  1.65s/it]
2024-08-07 20:12:52,887 - INFO: train loss: 0.02:  11%|#1        | 33/300 [00:58<07:29,  1.68s/it]
2024-08-07 20:12:57,814 - INFO: train loss: 0.01:  12%|#2        | 36/300 [01:03<07:21,  1.67s/it]
2024-08-07 20:13:02,513 - INFO: train loss: 0.01:  13%|#3        | 39/300 [01:08<07:07,  1.64s/it]
2024-08-07 20:13:07,348 - INFO: train loss: 0.00:  14%|#4        | 42/300 [01:13<07:00,  1.63s/it]
2024-08-07 20:13:12,463 - INFO: train loss: 0.01:  15%|#5        | 45/300 [01:18<07:01,  1.65s/it]
2024-08-07 20:13:17,454 - INFO: train loss: 0.02:  16%|#6        | 48/300 [01:23<06:57,  1.66s/it]
2024-08-07 20:13:22,641 - INFO: train loss: 0.02:  17%|#7        | 51/300 [01:28<06:57,  1.68s/it]
2024-08-07 20:13:27,472 - INFO: train loss: 0.02:  18%|#8        | 54/300 [01:33<06:47,  1.66s/it]
2024-08-07 20:13:32,480 - INFO: train loss: 0.00:  19%|#9        | 57/300 [01:38<06:43,  1.66s/it]
2024-08-07 20:13:37,073 - INFO: train loss: 0.01:  20%|##        | 60/300 [01:42<06:29,  1.62s/it]
2024-08-07 20:13:42,574 - INFO: train loss: 0.02:  21%|##1       | 63/300 [01:48<06:39,  1.69s/it]
2024-08-07 20:13:47,327 - INFO: train loss: 0.02:  22%|##2       | 66/300 [01:53<06:27,  1.66s/it]
2024-08-07 20:13:52,481 - INFO: train loss: 0.02:  23%|##3       | 69/300 [01:58<06:26,  1.67s/it]
2024-08-07 20:13:57,520 - INFO: train loss: 0.01:  24%|##4       | 72/300 [02:03<06:22,  1.68s/it]
2024-08-07 20:14:02,908 - INFO: train loss: 0.00:  25%|##5       | 75/300 [02:08<06:25,  1.71s/it]
2024-08-07 20:14:07,360 - INFO: train loss: 0.00:  26%|##6       | 78/300 [02:13<06:04,  1.64s/it]
2024-08-07 20:14:12,444 - INFO: train loss: 0.00:  27%|##7       | 81/300 [02:18<06:03,  1.66s/it]
2024-08-07 20:14:17,443 - INFO: train loss: 0.00:  28%|##8       | 84/300 [02:23<05:58,  1.66s/it]
2024-08-07 20:14:22,297 - INFO: train loss: 0.00:  29%|##9       | 87/300 [02:28<05:51,  1.65s/it]
2024-08-07 20:14:27,180 - INFO: train loss: 0.01:  30%|###       | 90/300 [02:32<05:44,  1.64s/it]
2024-08-07 20:14:31,597 - INFO: train loss: 0.01:  31%|###1      | 93/300 [02:37<05:29,  1.59s/it]
2024-08-07 20:14:36,922 - INFO: train loss: 0.01:  32%|###2      | 96/300 [02:42<05:35,  1.65s/it]
2024-08-07 20:14:41,847 - INFO: train loss: 0.00:  33%|###3      | 99/300 [02:47<05:30,  1.64s/it]
2024-08-07 20:14:47,270 - INFO: train loss: 0.01:  34%|###4      | 102/300 [02:52<05:35,  1.69s/it]
2024-08-07 20:14:52,339 - INFO: train loss: 0.01:  35%|###5      | 105/300 [02:58<05:30,  1.69s/it]
2024-08-07 20:14:57,282 - INFO: train loss: 0.01:  36%|###6      | 108/300 [03:03<05:22,  1.68s/it]
2024-08-07 20:15:01,768 - INFO: train loss: 0.01:  37%|###7      | 111/300 [03:07<05:06,  1.62s/it]
2024-08-07 20:15:07,162 - INFO: train loss: 0.01:  38%|###8      | 114/300 [03:12<05:11,  1.68s/it]
2024-08-07 20:15:12,529 - INFO: train loss: 0.02:  39%|###9      | 117/300 [03:18<05:12,  1.71s/it]
2024-08-07 20:15:17,943 - INFO: train loss: 0.02:  40%|####      | 120/300 [03:23<05:12,  1.74s/it]
2024-08-07 20:15:22,732 - INFO: train loss: 0.02:  41%|####1     | 123/300 [03:28<05:00,  1.70s/it]
2024-08-07 20:15:27,789 - INFO: train loss: 0.02:  42%|####2     | 126/300 [03:33<04:54,  1.69s/it]
2024-08-07 20:15:32,907 - INFO: train loss: 0.01:  43%|####3     | 129/300 [03:38<04:50,  1.70s/it]
2024-08-07 20:15:38,273 - INFO: train loss: 0.01:  44%|####4     | 132/300 [03:43<04:49,  1.72s/it]
2024-08-07 20:15:43,918 - INFO: train loss: 0.02:  45%|####5     | 135/300 [03:49<04:52,  1.77s/it]
2024-08-07 20:15:48,616 - INFO: train loss: 0.02:  46%|####6     | 138/300 [03:54<04:37,  1.71s/it]
2024-08-07 20:15:53,974 - INFO: train loss: 0.01:  47%|####6     | 141/300 [03:59<04:35,  1.73s/it]
2024-08-07 20:16:00,296 - INFO: train loss: 0.02:  48%|####8     | 144/300 [04:06<04:47,  1.85s/it]
2024-08-07 20:16:04,958 - INFO: train loss: 0.00:  49%|####9     | 147/300 [04:10<04:28,  1.76s/it]
2024-08-07 20:16:09,938 - INFO: train loss: 0.00:  50%|#####     | 150/300 [04:15<04:19,  1.73s/it]
2024-08-07 20:16:15,388 - INFO: train loss: 0.00:  51%|#####1    | 153/300 [04:21<04:17,  1.75s/it]
2024-08-07 20:16:20,326 - INFO: train loss: 0.00:  52%|#####2    | 156/300 [04:26<04:08,  1.72s/it]
2024-08-07 20:16:25,522 - INFO: train loss: 0.01:  53%|#####3    | 159/300 [04:31<04:03,  1.73s/it]
2024-08-07 20:16:30,797 - INFO: train loss: 0.01:  54%|#####4    | 162/300 [04:36<03:59,  1.74s/it]
2024-08-07 20:16:35,492 - INFO: train loss: 0.01:  55%|#####5    | 165/300 [04:41<03:47,  1.68s/it]
2024-08-07 20:16:40,745 - INFO: train loss: 0.01:  56%|#####6    | 168/300 [04:46<03:44,  1.70s/it]
2024-08-07 20:16:45,355 - INFO: train loss: 0.01:  57%|#####6    | 171/300 [04:51<03:33,  1.65s/it]
2024-08-07 20:16:50,156 - INFO: train loss: 0.01:  58%|#####8    | 174/300 [04:55<03:26,  1.64s/it]
2024-08-07 20:16:55,009 - INFO: train loss: 0.00:  59%|#####8    | 177/300 [05:00<03:20,  1.63s/it]
2024-08-07 20:16:59,859 - INFO: train loss: 0.01:  60%|######    | 180/300 [05:05<03:15,  1.63s/it]
2024-08-07 20:17:05,028 - INFO: train loss: 0.02:  61%|######1   | 183/300 [05:10<03:13,  1.66s/it]
2024-08-07 20:17:09,693 - INFO: train loss: 0.01:  62%|######2   | 186/300 [05:15<03:05,  1.63s/it]
2024-08-07 20:17:14,655 - INFO: train loss: 0.01:  63%|######3   | 189/300 [05:20<03:01,  1.63s/it]
2024-08-07 20:17:19,897 - INFO: train loss: 0.02:  64%|######4   | 192/300 [05:25<03:00,  1.67s/it]
2024-08-07 20:17:25,024 - INFO: train loss: 0.02:  65%|######5   | 195/300 [05:30<02:56,  1.68s/it]
2024-08-07 20:17:29,873 - INFO: train loss: 0.01:  66%|######6   | 198/300 [05:35<02:49,  1.66s/it]
2024-08-07 20:17:34,662 - INFO: train loss: 0.00:  67%|######7   | 201/300 [05:40<02:42,  1.64s/it]
2024-08-07 20:17:40,024 - INFO: train loss: 0.00:  68%|######8   | 204/300 [05:45<02:41,  1.69s/it]
2024-08-07 20:17:45,230 - INFO: train loss: 0.00:  69%|######9   | 207/300 [05:50<02:38,  1.70s/it]
2024-08-07 20:17:49,819 - INFO: train loss: 0.02:  70%|#######   | 210/300 [05:55<02:28,  1.65s/it]
2024-08-07 20:17:55,232 - INFO: train loss: 0.02:  71%|#######1  | 213/300 [06:00<02:27,  1.70s/it]
2024-08-07 20:18:00,534 - INFO: train loss: 0.02:  72%|#######2  | 216/300 [06:06<02:24,  1.72s/it]
2024-08-07 20:18:05,513 - INFO: train loss: 0.01:  73%|#######3  | 219/300 [06:11<02:17,  1.70s/it]
2024-08-07 20:18:09,898 - INFO: train loss: 0.00:  74%|#######4  | 222/300 [06:15<02:07,  1.63s/it]
2024-08-07 20:18:14,500 - INFO: train loss: 0.01:  75%|#######5  | 225/300 [06:20<02:00,  1.60s/it]
2024-08-07 20:18:19,901 - INFO: train loss: 0.01:  76%|#######6  | 228/300 [06:25<01:59,  1.66s/it]
2024-08-07 20:18:24,874 - INFO: train loss: 0.02:  77%|#######7  | 231/300 [06:30<01:54,  1.66s/it]
2024-08-07 20:18:30,311 - INFO: train loss: 0.03:  78%|#######8  | 234/300 [06:36<01:52,  1.71s/it]
2024-08-07 20:18:35,272 - INFO: train loss: 0.02:  79%|#######9  | 237/300 [06:40<01:46,  1.69s/it]
2024-08-07 20:18:39,913 - INFO: train loss: 0.01:  80%|########  | 240/300 [06:45<01:38,  1.65s/it]
2024-08-07 20:18:44,501 - INFO: train loss: 0.01:  81%|########1 | 243/300 [06:50<01:31,  1.61s/it]
2024-08-07 20:18:49,281 - INFO: train loss: 0.00:  82%|########2 | 246/300 [06:55<01:26,  1.61s/it]
2024-08-07 20:18:54,155 - INFO: train loss: 0.00:  83%|########2 | 249/300 [06:59<01:22,  1.61s/it]
2024-08-07 20:18:59,680 - INFO: train loss: 0.01:  84%|########4 | 252/300 [07:05<01:20,  1.68s/it]
2024-08-07 20:19:04,938 - INFO: train loss: 0.01:  85%|########5 | 255/300 [07:10<01:16,  1.70s/it]
2024-08-07 20:19:10,024 - INFO: train loss: 0.01:  86%|########6 | 258/300 [07:15<01:11,  1.70s/it]
2024-08-07 20:19:14,899 - INFO: train loss: 0.01:  87%|########7 | 261/300 [07:20<01:05,  1.68s/it]
2024-08-07 20:19:20,212 - INFO: train loss: 0.00:  88%|########8 | 264/300 [07:25<01:01,  1.71s/it]
2024-08-07 20:19:25,336 - INFO: train loss: 0.00:  89%|########9 | 267/300 [07:31<00:56,  1.71s/it]
2024-08-07 20:19:30,534 - INFO: train loss: 0.00:  90%|######### | 270/300 [07:36<00:51,  1.71s/it]
2024-08-07 20:19:35,732 - INFO: train loss: 0.00:  91%|#########1| 273/300 [07:41<00:46,  1.72s/it]
2024-08-07 20:19:40,862 - INFO: train loss: 0.00:  92%|#########2| 276/300 [07:46<00:41,  1.72s/it]
2024-08-07 20:19:45,387 - INFO: train loss: 0.00:  93%|#########3| 279/300 [07:51<00:34,  1.65s/it]
2024-08-07 20:19:50,372 - INFO: train loss: 0.01:  94%|#########3| 282/300 [07:56<00:29,  1.66s/it]
2024-08-07 20:19:55,253 - INFO: train loss: 0.01:  95%|#########5| 285/300 [08:00<00:24,  1.65s/it]
2024-08-07 20:20:00,260 - INFO: train loss: 0.01:  96%|#########6| 288/300 [08:05<00:19,  1.65s/it]
2024-08-07 20:20:05,453 - INFO: train loss: 0.01:  97%|#########7| 291/300 [08:11<00:15,  1.68s/it]
2024-08-07 20:20:10,743 - INFO: train loss: 0.01:  98%|#########8| 294/300 [08:16<00:10,  1.70s/it]
2024-08-07 20:20:16,042 - INFO: train loss: 0.03:  99%|#########9| 297/300 [08:21<00:05,  1.72s/it]
2024-08-07 20:20:20,700 - INFO: train loss: 0.04: 100%|##########| 300/300 [08:26<00:00,  1.67s/it]
2024-08-07 20:20:20,700 - INFO: train loss: 0.04: 100%|##########| 300/300 [08:26<00:00,  1.69s/it]
2024-08-07 20:20:20,700 - INFO: Saving last model checkpoint to /app/output
2024-08-07 20:20:20,700 - INFO: Saving checkpoint..
2024-08-07 20:20:47,550 - INFO: Starting validation inference
2024-08-07 20:20:47,551 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 20:20:52,956 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:09,  1.80s/it]
2024-08-07 20:20:54,473 - INFO: validation progress:   8%|8         | 6/75 [00:06<01:11,  1.04s/it]
2024-08-07 20:20:55,907 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:51,  1.28it/s]
2024-08-07 20:20:57,418 - INFO: validation progress:  16%|#6        | 12/75 [00:09<00:42,  1.49it/s]
2024-08-07 20:20:59,061 - INFO: validation progress:  20%|##        | 15/75 [00:11<00:37,  1.59it/s]
2024-08-07 20:21:00,620 - INFO: validation progress:  24%|##4       | 18/75 [00:13<00:33,  1.69it/s]
2024-08-07 20:21:02,302 - INFO: validation progress:  28%|##8       | 21/75 [00:14<00:31,  1.72it/s]
2024-08-07 20:21:03,796 - INFO: validation progress:  32%|###2      | 24/75 [00:16<00:28,  1.80it/s]
2024-08-07 20:21:05,438 - INFO: validation progress:  36%|###6      | 27/75 [00:17<00:26,  1.81it/s]
2024-08-07 20:21:07,031 - INFO: validation progress:  40%|####      | 30/75 [00:19<00:24,  1.83it/s]
2024-08-07 20:21:08,662 - INFO: validation progress:  44%|####4     | 33/75 [00:21<00:22,  1.83it/s]
2024-08-07 20:21:10,298 - INFO: validation progress:  48%|####8     | 36/75 [00:22<00:21,  1.83it/s]
2024-08-07 20:21:11,957 - INFO: validation progress:  52%|#####2    | 39/75 [00:24<00:19,  1.83it/s]
2024-08-07 20:21:13,531 - INFO: validation progress:  56%|#####6    | 42/75 [00:25<00:17,  1.85it/s]
2024-08-07 20:21:15,081 - INFO: validation progress:  60%|######    | 45/75 [00:27<00:16,  1.87it/s]
2024-08-07 20:21:16,537 - INFO: validation progress:  64%|######4   | 48/75 [00:28<00:14,  1.93it/s]
2024-08-07 20:21:18,046 - INFO: validation progress:  68%|######8   | 51/75 [00:30<00:12,  1.95it/s]
2024-08-07 20:21:19,623 - INFO: validation progress:  72%|#######2  | 54/75 [00:32<00:10,  1.93it/s]
2024-08-07 20:21:21,196 - INFO: validation progress:  76%|#######6  | 57/75 [00:33<00:09,  1.92it/s]
2024-08-07 20:21:22,636 - INFO: validation progress:  80%|########  | 60/75 [00:35<00:07,  1.97it/s]
2024-08-07 20:21:24,343 - INFO: validation progress:  84%|########4 | 63/75 [00:36<00:06,  1.90it/s]
2024-08-07 20:21:25,983 - INFO: validation progress:  88%|########8 | 66/75 [00:38<00:04,  1.88it/s]
2024-08-07 20:21:27,654 - INFO: validation progress:  92%|#########2| 69/75 [00:40<00:03,  1.85it/s]
2024-08-07 20:21:29,115 - INFO: validation progress:  96%|#########6| 72/75 [00:41<00:01,  1.91it/s]
2024-08-07 20:21:30,892 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.84it/s]
2024-08-07 20:21:30,895 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.73it/s]
2024-08-07 20:21:31,175 - INFO: Validation AUC: 0.98508
2024-08-07 20:21:31,175 - INFO: Mean validation loss: 0.64293
2024-08-07 20:21:31,522 - INFO: Training Epoch: 9 / 10
2024-08-07 20:21:31,522 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 20:21:35,983 - INFO: Evaluation step: 300
2024-08-07 20:21:41,085 - INFO: train loss: 0.03:   1%|1         | 3/300 [00:09<15:46,  3.19s/it]
2024-08-07 20:21:46,070 - INFO: train loss: 0.01:   2%|2         | 6/300 [00:14<11:13,  2.29s/it]
2024-08-07 20:21:51,064 - INFO: train loss: 0.01:   3%|3         | 9/300 [00:19<09:43,  2.00s/it]
2024-08-07 20:21:55,990 - INFO: train loss: 0.00:   4%|4         | 12/300 [00:24<08:56,  1.86s/it]
2024-08-07 20:22:00,464 - INFO: train loss: 0.00:   5%|5         | 15/300 [00:28<08:12,  1.73s/it]
2024-08-07 20:22:05,384 - INFO: train loss: 0.00:   6%|6         | 18/300 [00:33<07:58,  1.70s/it]
2024-08-07 20:22:10,373 - INFO: train loss: 0.00:   7%|7         | 21/300 [00:38<07:50,  1.69s/it]
2024-08-07 20:22:15,589 - INFO: train loss: 0.00:   8%|8         | 24/300 [00:44<07:50,  1.70s/it]
2024-08-07 20:22:20,810 - INFO: train loss: 0.00:   9%|9         | 27/300 [00:49<07:48,  1.71s/it]
2024-08-07 20:22:26,006 - INFO: train loss: 0.00:  10%|#         | 30/300 [00:54<07:44,  1.72s/it]
2024-08-07 20:22:31,261 - INFO: train loss: 0.00:  11%|#1        | 33/300 [00:59<07:41,  1.73s/it]
2024-08-07 20:22:36,122 - INFO: train loss: 0.00:  12%|#2        | 36/300 [01:04<07:27,  1.70s/it]
2024-08-07 20:22:40,522 - INFO: train loss: 0.00:  13%|#3        | 39/300 [01:08<07:04,  1.63s/it]
2024-08-07 20:22:45,484 - INFO: train loss: 0.00:  14%|#4        | 42/300 [01:13<07:01,  1.64s/it]
2024-08-07 20:22:50,381 - INFO: train loss: 0.00:  15%|#5        | 45/300 [01:18<06:56,  1.63s/it]
2024-08-07 20:22:55,456 - INFO: train loss: 0.00:  16%|#6        | 48/300 [01:23<06:56,  1.65s/it]
2024-08-07 20:23:00,381 - INFO: train loss: 0.00:  17%|#7        | 51/300 [01:28<06:50,  1.65s/it]
2024-08-07 20:23:05,319 - INFO: train loss: 0.00:  18%|#8        | 54/300 [01:33<06:45,  1.65s/it]
2024-08-07 20:23:10,174 - INFO: train loss: 0.00:  19%|#9        | 57/300 [01:38<06:38,  1.64s/it]
2024-08-07 20:23:15,054 - INFO: train loss: 0.00:  20%|##        | 60/300 [01:43<06:32,  1.64s/it]
2024-08-07 20:23:20,225 - INFO: train loss: 0.00:  21%|##1       | 63/300 [01:48<06:33,  1.66s/it]
2024-08-07 20:23:25,740 - INFO: train loss: 0.00:  22%|##2       | 66/300 [01:54<06:41,  1.71s/it]
2024-08-07 20:23:30,691 - INFO: train loss: 0.00:  23%|##3       | 69/300 [01:59<06:31,  1.70s/it]
2024-08-07 20:23:35,634 - INFO: train loss: 0.00:  24%|##4       | 72/300 [02:04<06:23,  1.68s/it]
2024-08-07 20:23:40,599 - INFO: train loss: 0.00:  25%|##5       | 75/300 [02:09<06:16,  1.67s/it]
2024-08-07 20:23:45,461 - INFO: train loss: 0.00:  26%|##6       | 78/300 [02:13<06:07,  1.66s/it]
2024-08-07 20:23:50,584 - INFO: train loss: 0.00:  27%|##7       | 81/300 [02:19<06:06,  1.67s/it]
2024-08-07 20:23:55,339 - INFO: train loss: 0.00:  28%|##8       | 84/300 [02:23<05:55,  1.65s/it]
2024-08-07 20:23:59,954 - INFO: train loss: 0.00:  29%|##9       | 87/300 [02:28<05:43,  1.61s/it]
2024-08-07 20:24:05,052 - INFO: train loss: 0.00:  30%|###       | 90/300 [02:33<05:44,  1.64s/it]
2024-08-07 20:24:10,095 - INFO: train loss: 0.00:  31%|###1      | 93/300 [02:38<05:41,  1.65s/it]
2024-08-07 20:24:15,656 - INFO: train loss: 0.00:  32%|###2      | 96/300 [02:44<05:49,  1.71s/it]
2024-08-07 20:24:20,679 - INFO: train loss: 0.00:  33%|###3      | 99/300 [02:49<05:41,  1.70s/it]
2024-08-07 20:24:25,731 - INFO: train loss: 0.00:  34%|###4      | 102/300 [02:54<05:35,  1.70s/it]
2024-08-07 20:24:30,595 - INFO: train loss: 0.00:  35%|###5      | 105/300 [02:59<05:26,  1.67s/it]
2024-08-07 20:24:35,802 - INFO: train loss: 0.00:  36%|###6      | 108/300 [03:04<05:24,  1.69s/it]
2024-08-07 20:24:41,025 - INFO: train loss: 0.00:  37%|###7      | 111/300 [03:09<05:22,  1.71s/it]
2024-08-07 20:24:46,397 - INFO: train loss: 0.01:  38%|###8      | 114/300 [03:14<05:22,  1.73s/it]
2024-08-07 20:24:51,763 - INFO: train loss: 0.01:  39%|###9      | 117/300 [03:20<05:20,  1.75s/it]
2024-08-07 20:24:56,472 - INFO: train loss: 0.01:  40%|####      | 120/300 [03:24<05:05,  1.70s/it]
2024-08-07 20:25:01,294 - INFO: train loss: 0.00:  41%|####1     | 123/300 [03:29<04:55,  1.67s/it]
2024-08-07 20:25:06,422 - INFO: train loss: 0.00:  42%|####2     | 126/300 [03:34<04:52,  1.68s/it]
2024-08-07 20:25:11,324 - INFO: train loss: 0.00:  43%|####3     | 129/300 [03:39<04:45,  1.67s/it]
2024-08-07 20:25:16,582 - INFO: train loss: 0.00:  44%|####4     | 132/300 [03:45<04:44,  1.69s/it]
2024-08-07 20:25:21,783 - INFO: train loss: 0.00:  45%|####5     | 135/300 [03:50<04:41,  1.70s/it]
2024-08-07 20:25:26,857 - INFO: train loss: 0.00:  46%|####6     | 138/300 [03:55<04:35,  1.70s/it]
2024-08-07 20:25:31,802 - INFO: train loss: 0.01:  47%|####6     | 141/300 [04:00<04:27,  1.69s/it]
2024-08-07 20:25:36,498 - INFO: train loss: 0.01:  48%|####8     | 144/300 [04:04<04:17,  1.65s/it]
2024-08-07 20:25:41,315 - INFO: train loss: 0.01:  49%|####9     | 147/300 [04:09<04:10,  1.64s/it]
2024-08-07 20:25:46,328 - INFO: train loss: 0.00:  50%|#####     | 150/300 [04:14<04:06,  1.65s/it]
2024-08-07 20:25:51,604 - INFO: train loss: 0.00:  51%|#####1    | 153/300 [04:20<04:06,  1.68s/it]
2024-08-07 20:25:56,342 - INFO: train loss: 0.00:  52%|#####2    | 156/300 [04:24<03:57,  1.65s/it]
2024-08-07 20:26:01,821 - INFO: train loss: 0.00:  53%|#####3    | 159/300 [04:30<04:00,  1.70s/it]
2024-08-07 20:26:07,553 - INFO: train loss: 0.00:  54%|#####4    | 162/300 [04:36<04:03,  1.77s/it]
2024-08-07 20:26:12,182 - INFO: train loss: 0.00:  55%|#####5    | 165/300 [04:40<03:49,  1.70s/it]
2024-08-07 20:26:17,532 - INFO: train loss: 0.00:  56%|#####6    | 168/300 [04:46<03:47,  1.72s/it]
2024-08-07 20:26:23,050 - INFO: train loss: 0.00:  57%|#####6    | 171/300 [04:51<03:46,  1.76s/it]
2024-08-07 20:26:28,175 - INFO: train loss: 0.00:  58%|#####8    | 174/300 [04:56<03:39,  1.74s/it]
2024-08-07 20:26:33,293 - INFO: train loss: 0.00:  59%|#####8    | 177/300 [05:01<03:33,  1.73s/it]
2024-08-07 20:26:38,011 - INFO: train loss: 0.00:  60%|######    | 180/300 [05:06<03:22,  1.68s/it]
2024-08-07 20:26:42,706 - INFO: train loss: 0.00:  61%|######1   | 183/300 [05:11<03:12,  1.65s/it]
2024-08-07 20:26:47,159 - INFO: train loss: 0.00:  62%|######2   | 186/300 [05:15<03:02,  1.60s/it]
2024-08-07 20:26:51,986 - INFO: train loss: 0.00:  63%|######3   | 189/300 [05:20<02:57,  1.60s/it]
2024-08-07 20:26:56,988 - INFO: train loss: 0.00:  64%|######4   | 192/300 [05:25<02:55,  1.62s/it]
2024-08-07 20:27:02,143 - INFO: train loss: 0.00:  65%|######5   | 195/300 [05:30<02:53,  1.65s/it]
2024-08-07 20:27:07,318 - INFO: train loss: 0.00:  66%|######6   | 198/300 [05:35<02:50,  1.67s/it]
2024-08-07 20:27:12,563 - INFO: train loss: 0.00:  67%|######7   | 201/300 [05:41<02:47,  1.70s/it]
2024-08-07 20:27:17,271 - INFO: train loss: 0.00:  68%|######8   | 204/300 [05:45<02:39,  1.66s/it]
2024-08-07 20:27:21,565 - INFO: train loss: 0.00:  69%|######9   | 207/300 [05:50<02:27,  1.59s/it]
2024-08-07 20:27:26,687 - INFO: train loss: 0.00:  70%|#######   | 210/300 [05:55<02:26,  1.63s/it]
2024-08-07 20:27:31,823 - INFO: train loss: 0.00:  71%|#######1  | 213/300 [06:00<02:23,  1.65s/it]
2024-08-07 20:27:36,961 - INFO: train loss: 0.00:  72%|#######2  | 216/300 [06:05<02:20,  1.67s/it]
2024-08-07 20:27:41,879 - INFO: train loss: 0.00:  73%|#######3  | 219/300 [06:10<02:14,  1.66s/it]
2024-08-07 20:27:46,529 - INFO: train loss: 0.00:  74%|#######4  | 222/300 [06:15<02:06,  1.63s/it]
2024-08-07 20:27:51,831 - INFO: train loss: 0.00:  75%|#######5  | 225/300 [06:20<02:05,  1.67s/it]
2024-08-07 20:27:56,966 - INFO: train loss: 0.00:  76%|#######6  | 228/300 [06:25<02:01,  1.68s/it]
2024-08-07 20:28:01,999 - INFO: train loss: 0.00:  77%|#######7  | 231/300 [06:30<01:55,  1.68s/it]
2024-08-07 20:28:06,994 - INFO: train loss: 0.00:  78%|#######8  | 234/300 [06:35<01:50,  1.68s/it]
2024-08-07 20:28:12,321 - INFO: train loss: 0.00:  79%|#######9  | 237/300 [06:40<01:47,  1.71s/it]
2024-08-07 20:28:18,126 - INFO: train loss: 0.00:  80%|########  | 240/300 [06:46<01:46,  1.77s/it]
2024-08-07 20:28:23,306 - INFO: train loss: 0.00:  81%|########1 | 243/300 [06:51<01:40,  1.76s/it]
2024-08-07 20:28:27,880 - INFO: train loss: 0.00:  82%|########2 | 246/300 [06:56<01:31,  1.69s/it]
2024-08-07 20:28:32,786 - INFO: train loss: 0.00:  83%|########2 | 249/300 [07:01<01:25,  1.67s/it]
2024-08-07 20:28:37,605 - INFO: train loss: 0.00:  84%|########4 | 252/300 [07:06<01:19,  1.65s/it]
2024-08-07 20:28:42,333 - INFO: train loss: 0.00:  85%|########5 | 255/300 [07:10<01:13,  1.63s/it]
2024-08-07 20:28:47,654 - INFO: train loss: 0.00:  86%|########6 | 258/300 [07:16<01:10,  1.67s/it]
2024-08-07 20:28:52,363 - INFO: train loss: 0.00:  87%|########7 | 261/300 [07:20<01:04,  1.64s/it]
2024-08-07 20:28:56,865 - INFO: train loss: 0.00:  88%|########8 | 264/300 [07:25<00:57,  1.60s/it]
2024-08-07 20:29:01,962 - INFO: train loss: 0.00:  89%|########9 | 267/300 [07:30<00:53,  1.63s/it]
2024-08-07 20:29:06,987 - INFO: train loss: 0.00:  90%|######### | 270/300 [07:35<00:49,  1.64s/it]
2024-08-07 20:29:12,440 - INFO: train loss: 0.00:  91%|#########1| 273/300 [07:40<00:45,  1.70s/it]
2024-08-07 20:29:17,750 - INFO: train loss: 0.00:  92%|#########2| 276/300 [07:46<00:41,  1.72s/it]
2024-08-07 20:29:22,608 - INFO: train loss: 0.00:  93%|#########3| 279/300 [07:51<00:35,  1.69s/it]
2024-08-07 20:29:28,596 - INFO: train loss: 0.00:  94%|#########3| 282/300 [07:57<00:32,  1.78s/it]
2024-08-07 20:29:34,487 - INFO: train loss: 0.00:  95%|#########5| 285/300 [08:02<00:27,  1.84s/it]
2024-08-07 20:29:39,896 - INFO: train loss: 0.00:  96%|#########6| 288/300 [08:08<00:21,  1.83s/it]
2024-08-07 20:29:44,719 - INFO: train loss: 0.00:  97%|#########7| 291/300 [08:13<00:15,  1.76s/it]
2024-08-07 20:29:49,663 - INFO: train loss: 0.00:  98%|#########8| 294/300 [08:18<00:10,  1.73s/it]
2024-08-07 20:29:54,606 - INFO: train loss: 0.00:  99%|#########9| 297/300 [08:23<00:05,  1.70s/it]
2024-08-07 20:29:59,956 - INFO: train loss: 0.00: 100%|##########| 300/300 [08:28<00:00,  1.73s/it]
2024-08-07 20:29:59,956 - INFO: train loss: 0.00: 100%|##########| 300/300 [08:28<00:00,  1.69s/it]
2024-08-07 20:29:59,956 - INFO: Saving last model checkpoint to /app/output
2024-08-07 20:29:59,957 - INFO: Saving checkpoint..
2024-08-07 20:30:29,425 - INFO: Starting validation inference
2024-08-07 20:30:29,426 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 20:30:34,701 - INFO: validation progress:   4%|4         | 3/75 [00:05<02:06,  1.76s/it]
2024-08-07 20:30:36,218 - INFO: validation progress:   8%|8         | 6/75 [00:06<01:10,  1.02s/it]
2024-08-07 20:30:37,655 - INFO: validation progress:  12%|#2        | 9/75 [00:08<00:51,  1.29it/s]
2024-08-07 20:30:39,155 - INFO: validation progress:  16%|#6        | 12/75 [00:09<00:41,  1.50it/s]
2024-08-07 20:30:40,762 - INFO: validation progress:  20%|##        | 15/75 [00:11<00:37,  1.62it/s]
2024-08-07 20:30:42,319 - INFO: validation progress:  24%|##4       | 18/75 [00:12<00:33,  1.71it/s]
2024-08-07 20:30:43,951 - INFO: validation progress:  28%|##8       | 21/75 [00:14<00:30,  1.75it/s]
2024-08-07 20:30:45,447 - INFO: validation progress:  32%|###2      | 24/75 [00:16<00:27,  1.82it/s]
2024-08-07 20:30:47,091 - INFO: validation progress:  36%|###6      | 27/75 [00:17<00:26,  1.82it/s]
2024-08-07 20:30:48,675 - INFO: validation progress:  40%|####      | 30/75 [00:19<00:24,  1.85it/s]
2024-08-07 20:30:50,310 - INFO: validation progress:  44%|####4     | 33/75 [00:20<00:22,  1.84it/s]
2024-08-07 20:30:51,946 - INFO: validation progress:  48%|####8     | 36/75 [00:22<00:21,  1.84it/s]
2024-08-07 20:30:53,607 - INFO: validation progress:  52%|#####2    | 39/75 [00:24<00:19,  1.83it/s]
2024-08-07 20:30:55,181 - INFO: validation progress:  56%|#####6    | 42/75 [00:25<00:17,  1.85it/s]
2024-08-07 20:30:56,732 - INFO: validation progress:  60%|######    | 45/75 [00:27<00:15,  1.88it/s]
2024-08-07 20:30:58,204 - INFO: validation progress:  64%|######4   | 48/75 [00:28<00:14,  1.92it/s]
2024-08-07 20:30:59,710 - INFO: validation progress:  68%|######8   | 51/75 [00:30<00:12,  1.94it/s]
2024-08-07 20:31:01,769 - INFO: validation progress:  72%|#######2  | 54/75 [00:32<00:11,  1.77it/s]
2024-08-07 20:31:03,388 - INFO: validation progress:  76%|#######6  | 57/75 [00:33<00:10,  1.79it/s]
2024-08-07 20:31:04,843 - INFO: validation progress:  80%|########  | 60/75 [00:35<00:08,  1.86it/s]
2024-08-07 20:31:06,549 - INFO: validation progress:  84%|########4 | 63/75 [00:37<00:06,  1.83it/s]
2024-08-07 20:31:08,186 - INFO: validation progress:  88%|########8 | 66/75 [00:38<00:04,  1.83it/s]
2024-08-07 20:31:09,819 - INFO: validation progress:  92%|#########2| 69/75 [00:40<00:03,  1.83it/s]
2024-08-07 20:31:11,283 - INFO: validation progress:  96%|#########6| 72/75 [00:41<00:01,  1.89it/s]
2024-08-07 20:31:13,013 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.84it/s]
2024-08-07 20:31:13,016 - INFO: validation progress: 100%|##########| 75/75 [00:43<00:00,  1.72it/s]
2024-08-07 20:31:13,295 - INFO: Validation AUC: 0.98502
2024-08-07 20:31:13,295 - INFO: Mean validation loss: 0.62766
2024-08-07 20:31:13,655 - INFO: Training Epoch: 10 / 10
2024-08-07 20:31:13,655 - INFO: train loss:   0%|          | 0/300 [00:00<?, ?it/s]
2024-08-07 20:31:17,753 - INFO: Evaluation step: 300
2024-08-07 20:31:22,773 - INFO: train loss: 0.00:   1%|1         | 3/300 [00:09<15:02,  3.04s/it]
2024-08-07 20:31:27,855 - INFO: train loss: 0.00:   2%|2         | 6/300 [00:14<11:00,  2.25s/it]
2024-08-07 20:31:34,058 - INFO: train loss: 0.00:   3%|3         | 9/300 [00:20<10:30,  2.17s/it]
2024-08-07 20:31:38,902 - INFO: train loss: 0.00:   4%|4         | 12/300 [00:25<09:21,  1.95s/it]
2024-08-07 20:31:44,119 - INFO: train loss: 0.00:   5%|5         | 15/300 [00:30<08:53,  1.87s/it]
2024-08-07 20:31:49,123 - INFO: train loss: 0.00:   6%|6         | 18/300 [00:35<08:28,  1.80s/it]
2024-08-07 20:31:53,861 - INFO: train loss: 0.00:   7%|7         | 21/300 [00:40<08:02,  1.73s/it]
2024-08-07 20:31:58,863 - INFO: train loss: 0.00:   8%|8         | 24/300 [00:45<07:51,  1.71s/it]
2024-08-07 20:32:03,567 - INFO: train loss: 0.00:   9%|9         | 27/300 [00:49<07:34,  1.67s/it]
2024-08-07 20:32:08,541 - INFO: train loss: 0.00:  10%|#         | 30/300 [00:54<07:29,  1.66s/it]
2024-08-07 20:32:13,658 - INFO: train loss: 0.00:  11%|#1        | 33/300 [01:00<07:27,  1.68s/it]
2024-08-07 20:32:18,172 - INFO: train loss: 0.00:  12%|#2        | 36/300 [01:04<07:08,  1.62s/it]
2024-08-07 20:32:23,681 - INFO: train loss: 0.00:  13%|#3        | 39/300 [01:10<07:20,  1.69s/it]
2024-08-07 20:32:28,564 - INFO: train loss: 0.00:  14%|#4        | 42/300 [01:14<07:10,  1.67s/it]
2024-08-07 20:32:33,779 - INFO: train loss: 0.00:  15%|#5        | 45/300 [01:20<07:11,  1.69s/it]
2024-08-07 20:32:38,734 - INFO: train loss: 0.00:  16%|#6        | 48/300 [01:25<07:03,  1.68s/it]
2024-08-07 20:32:43,643 - INFO: train loss: 0.00:  17%|#7        | 51/300 [01:29<06:54,  1.67s/it]
2024-08-07 20:32:49,121 - INFO: train loss: 0.00:  18%|#8        | 54/300 [01:35<07:01,  1.71s/it]
2024-08-07 20:32:54,109 - INFO: train loss: 0.00:  19%|#9        | 57/300 [01:40<06:52,  1.70s/it]
2024-08-07 20:32:59,083 - INFO: train loss: 0.00:  20%|##        | 60/300 [01:45<06:44,  1.69s/it]
2024-08-07 20:33:03,709 - INFO: train loss: 0.00:  21%|##1       | 63/300 [01:50<06:29,  1.64s/it]
2024-08-07 20:33:08,781 - INFO: train loss: 0.00:  22%|##2       | 66/300 [01:55<06:27,  1.66s/it]
2024-08-07 20:33:13,989 - INFO: train loss: 0.00:  23%|##3       | 69/300 [02:00<06:28,  1.68s/it]
2024-08-07 20:33:18,529 - INFO: train loss: 0.00:  24%|##4       | 72/300 [02:04<06:11,  1.63s/it]
2024-08-07 20:33:23,203 - INFO: train loss: 0.00:  25%|##5       | 75/300 [02:09<06:01,  1.61s/it]
2024-08-07 20:33:28,215 - INFO: train loss: 0.00:  26%|##6       | 78/300 [02:14<06:01,  1.63s/it]
2024-08-07 20:33:33,197 - INFO: train loss: 0.00:  27%|##7       | 81/300 [02:19<05:58,  1.64s/it]
2024-08-07 20:33:38,164 - INFO: train loss: 0.00:  28%|##8       | 84/300 [02:24<05:54,  1.64s/it]
2024-08-07 20:33:43,257 - INFO: train loss: 0.00:  29%|##9       | 87/300 [02:29<05:53,  1.66s/it]
2024-08-07 20:33:48,455 - INFO: train loss: 0.00:  30%|###       | 90/300 [02:34<05:53,  1.68s/it]
2024-08-07 20:33:53,404 - INFO: train loss: 0.00:  31%|###1      | 93/300 [02:39<05:46,  1.67s/it]
2024-08-07 20:33:58,486 - INFO: train loss: 0.00:  32%|###2      | 96/300 [02:44<05:42,  1.68s/it]
2024-08-07 20:34:03,474 - INFO: train loss: 0.00:  33%|###3      | 99/300 [02:49<05:36,  1.67s/it]
2024-08-07 20:34:08,411 - INFO: train loss: 0.00:  34%|###4      | 102/300 [02:54<05:29,  1.67s/it]
2024-08-07 20:34:13,419 - INFO: train loss: 0.00:  35%|###5      | 105/300 [02:59<05:24,  1.67s/it]
2024-08-07 20:34:18,516 - INFO: train loss: 0.00:  36%|###6      | 108/300 [03:04<05:21,  1.68s/it]
2024-08-07 20:34:23,360 - INFO: train loss: 0.00:  37%|###7      | 111/300 [03:09<05:13,  1.66s/it]
2024-08-07 20:34:28,372 - INFO: train loss: 0.00:  38%|###8      | 114/300 [03:14<05:09,  1.66s/it]
2024-08-07 20:34:33,621 - INFO: train loss: 0.00:  39%|###9      | 117/300 [03:19<05:08,  1.69s/it]
2024-08-07 20:34:38,503 - INFO: train loss: 0.00:  40%|####      | 120/300 [03:24<05:00,  1.67s/it]
2024-08-07 20:34:43,505 - INFO: train loss: 0.00:  41%|####1     | 123/300 [03:29<04:55,  1.67s/it]
2024-08-07 20:34:48,661 - INFO: train loss: 0.00:  42%|####2     | 126/300 [03:35<04:53,  1.68s/it]
2024-08-07 20:34:53,573 - INFO: train loss: 0.00:  43%|####3     | 129/300 [03:39<04:45,  1.67s/it]
2024-08-07 20:34:59,045 - INFO: train loss: 0.00:  44%|####4     | 132/300 [03:45<04:48,  1.72s/it]
2024-08-07 20:35:04,183 - INFO: train loss: 0.00:  45%|####5     | 135/300 [03:50<04:42,  1.72s/it]
2024-08-07 20:35:09,347 - INFO: train loss: 0.00:  46%|####6     | 138/300 [03:55<04:38,  1.72s/it]
2024-08-07 20:35:14,334 - INFO: train loss: 0.00:  47%|####6     | 141/300 [04:00<04:30,  1.70s/it]
2024-08-07 20:35:19,823 - INFO: train loss: 0.00:  48%|####8     | 144/300 [04:06<04:31,  1.74s/it]
2024-08-07 20:35:24,659 - INFO: train loss: 0.00:  49%|####9     | 147/300 [04:11<04:20,  1.70s/it]
2024-08-07 20:35:30,031 - INFO: train loss: 0.00:  50%|#####     | 150/300 [04:16<04:19,  1.73s/it]
2024-08-07 20:35:34,793 - INFO: train loss: 0.00:  51%|#####1    | 153/300 [04:21<04:07,  1.69s/it]
2024-08-07 20:35:39,878 - INFO: train loss: 0.00:  52%|#####2    | 156/300 [04:26<04:03,  1.69s/it]
2024-08-07 20:35:44,837 - INFO: train loss: 0.00:  53%|#####3    | 159/300 [04:31<03:56,  1.68s/it]
2024-08-07 20:35:49,843 - INFO: train loss: 0.00:  54%|#####4    | 162/300 [04:36<03:51,  1.68s/it]
2024-08-07 20:35:54,615 - INFO: train loss: 0.00:  55%|#####5    | 165/300 [04:40<03:42,  1.65s/it]
2024-08-07 20:35:59,471 - INFO: train loss: 0.00:  56%|#####6    | 168/300 [04:45<03:36,  1.64s/it]
2024-08-07 20:36:04,296 - INFO: train loss: 0.00:  57%|#####6    | 171/300 [04:50<03:30,  1.63s/it]
2024-08-07 20:36:09,285 - INFO: train loss: 0.00:  58%|#####8    | 174/300 [04:55<03:26,  1.64s/it]
2024-08-07 20:36:13,895 - INFO: train loss: 0.00:  59%|#####8    | 177/300 [05:00<03:17,  1.61s/it]
2024-08-07 20:36:18,875 - INFO: train loss: 0.00:  60%|######    | 180/300 [05:05<03:14,  1.62s/it]
2024-08-07 20:36:23,555 - INFO: train loss: 0.00:  61%|######1   | 183/300 [05:09<03:07,  1.61s/it]
2024-08-07 20:36:28,927 - INFO: train loss: 0.00:  62%|######2   | 186/300 [05:15<03:09,  1.66s/it]
2024-08-07 20:36:34,270 - INFO: train loss: 0.00:  63%|######3   | 189/300 [05:20<03:08,  1.70s/it]
2024-08-07 20:36:38,927 - INFO: train loss: 0.00:  64%|######4   | 192/300 [05:25<02:58,  1.65s/it]
2024-08-07 20:36:44,145 - INFO: train loss: 0.00:  65%|######5   | 195/300 [05:30<02:56,  1.68s/it]
2024-08-07 20:36:49,154 - INFO: train loss: 0.00:  66%|######6   | 198/300 [05:35<02:50,  1.68s/it]
2024-08-07 20:36:53,844 - INFO: train loss: 0.00:  67%|######7   | 201/300 [05:40<02:42,  1.64s/it]
2024-08-07 20:37:00,123 - INFO: train loss: 0.00:  68%|######8   | 204/300 [05:46<02:50,  1.78s/it]
2024-08-07 20:37:05,665 - INFO: train loss: 0.00:  69%|######9   | 207/300 [05:52<02:47,  1.80s/it]
2024-08-07 20:37:10,768 - INFO: train loss: 0.00:  70%|#######   | 210/300 [05:57<02:39,  1.77s/it]
2024-08-07 20:37:15,538 - INFO: train loss: 0.00:  71%|#######1  | 213/300 [06:01<02:29,  1.72s/it]
2024-08-07 20:37:20,535 - INFO: train loss: 0.00:  72%|#######2  | 216/300 [06:06<02:22,  1.70s/it]
2024-08-07 20:37:25,562 - INFO: train loss: 0.00:  73%|#######3  | 219/300 [06:11<02:17,  1.69s/it]
2024-08-07 20:37:30,959 - INFO: train loss: 0.00:  74%|#######4  | 222/300 [06:17<02:14,  1.72s/it]
2024-08-07 20:37:36,096 - INFO: train loss: 0.00:  75%|#######5  | 225/300 [06:22<02:09,  1.72s/it]
2024-08-07 20:37:41,524 - INFO: train loss: 0.00:  76%|#######6  | 228/300 [06:27<02:05,  1.75s/it]
2024-08-07 20:37:46,362 - INFO: train loss: 0.00:  77%|#######7  | 231/300 [06:32<01:57,  1.71s/it]
2024-08-07 20:37:51,208 - INFO: train loss: 0.00:  78%|#######8  | 234/300 [06:37<01:50,  1.68s/it]
2024-08-07 20:37:56,321 - INFO: train loss: 0.00:  79%|#######9  | 237/300 [06:42<01:46,  1.69s/it]
2024-08-07 20:38:01,325 - INFO: train loss: 0.00:  80%|########  | 240/300 [06:47<01:40,  1.68s/it]
2024-08-07 20:38:06,370 - INFO: train loss: 0.00:  81%|########1 | 243/300 [06:52<01:35,  1.68s/it]
2024-08-07 20:38:11,090 - INFO: train loss: 0.00:  82%|########2 | 246/300 [06:57<01:29,  1.65s/it]
2024-08-07 20:38:16,276 - INFO: train loss: 0.00:  83%|########2 | 249/300 [07:02<01:25,  1.67s/it]
2024-08-07 20:38:21,081 - INFO: train loss: 0.00:  84%|########4 | 252/300 [07:07<01:19,  1.65s/it]
2024-08-07 20:38:26,445 - INFO: train loss: 0.00:  85%|########5 | 255/300 [07:12<01:16,  1.69s/it]
2024-08-07 20:38:31,447 - INFO: train loss: 0.00:  86%|########6 | 258/300 [07:17<01:10,  1.68s/it]
2024-08-07 20:38:36,844 - INFO: train loss: 0.00:  87%|########7 | 261/300 [07:23<01:07,  1.72s/it]
2024-08-07 20:38:41,909 - INFO: train loss: 0.00:  88%|########8 | 264/300 [07:28<01:01,  1.71s/it]
2024-08-07 20:38:46,957 - INFO: train loss: 0.00:  89%|########9 | 267/300 [07:33<00:56,  1.70s/it]
2024-08-07 20:38:51,537 - INFO: train loss: 0.00:  90%|######### | 270/300 [07:37<00:49,  1.65s/it]
2024-08-07 20:38:56,346 - INFO: train loss: 0.00:  91%|#########1| 273/300 [07:42<00:44,  1.64s/it]
2024-08-07 20:39:01,231 - INFO: train loss: 0.00:  92%|#########2| 276/300 [07:47<00:39,  1.63s/it]
2024-08-07 20:39:05,855 - INFO: train loss: 0.00:  93%|#########3| 279/300 [07:52<00:33,  1.61s/it]
2024-08-07 20:39:10,535 - INFO: train loss: 0.00:  94%|#########3| 282/300 [07:56<00:28,  1.59s/it]
2024-08-07 20:39:15,560 - INFO: train loss: 0.00:  95%|#########5| 285/300 [08:01<00:24,  1.62s/it]
2024-08-07 20:39:20,898 - INFO: train loss: 0.00:  96%|#########6| 288/300 [08:07<00:19,  1.67s/it]
2024-08-07 20:39:25,862 - INFO: train loss: 0.00:  97%|#########7| 291/300 [08:12<00:14,  1.66s/it]
2024-08-07 20:39:30,817 - INFO: train loss: 0.00:  98%|#########8| 294/300 [08:17<00:09,  1.66s/it]
2024-08-07 20:39:36,277 - INFO: train loss: 0.00:  99%|#########9| 297/300 [08:22<00:05,  1.71s/it]
2024-08-07 20:39:40,891 - INFO: train loss: 0.00: 100%|##########| 300/300 [08:27<00:00,  1.66s/it]
2024-08-07 20:39:40,894 - INFO: train loss: 0.00: 100%|##########| 300/300 [08:27<00:00,  1.69s/it]
2024-08-07 20:39:40,894 - INFO: Saving last model checkpoint to /app/output
2024-08-07 20:39:40,894 - INFO: Saving checkpoint..
2024-08-07 20:40:08,676 - INFO: Starting validation inference
2024-08-07 20:40:08,678 - INFO: validation progress:   0%|          | 0/75 [00:00<?, ?it/s]
2024-08-07 20:40:15,254 - INFO: validation progress:   4%|4         | 3/75 [00:06<02:37,  2.19s/it]
2024-08-07 20:40:16,775 - INFO: validation progress:   8%|8         | 6/75 [00:08<01:22,  1.20s/it]
2024-08-07 20:40:18,262 - INFO: validation progress:  12%|#2        | 9/75 [00:09<00:58,  1.14it/s]
2024-08-07 20:40:19,824 - INFO: validation progress:  16%|#6        | 12/75 [00:11<00:46,  1.36it/s]
2024-08-07 20:40:21,434 - INFO: validation progress:  20%|##        | 15/75 [00:12<00:39,  1.50it/s]
2024-08-07 20:40:22,999 - INFO: validation progress:  24%|##4       | 18/75 [00:14<00:35,  1.62it/s]
2024-08-07 20:40:24,632 - INFO: validation progress:  28%|##8       | 21/75 [00:15<00:32,  1.69it/s]
2024-08-07 20:40:26,126 - INFO: validation progress:  32%|###2      | 24/75 [00:17<00:28,  1.78it/s]
2024-08-07 20:40:27,817 - INFO: validation progress:  36%|###6      | 27/75 [00:19<00:27,  1.78it/s]
2024-08-07 20:40:29,457 - INFO: validation progress:  40%|####      | 30/75 [00:20<00:25,  1.79it/s]
2024-08-07 20:40:31,090 - INFO: validation progress:  44%|####4     | 33/75 [00:22<00:23,  1.81it/s]
2024-08-07 20:40:32,727 - INFO: validation progress:  48%|####8     | 36/75 [00:24<00:21,  1.81it/s]
2024-08-07 20:40:34,393 - INFO: validation progress:  52%|#####2    | 39/75 [00:25<00:19,  1.81it/s]
2024-08-07 20:40:35,972 - INFO: validation progress:  56%|#####6    | 42/75 [00:27<00:17,  1.84it/s]
2024-08-07 20:40:37,524 - INFO: validation progress:  60%|######    | 45/75 [00:28<00:16,  1.86it/s]
2024-08-07 20:40:38,989 - INFO: validation progress:  64%|######4   | 48/75 [00:30<00:14,  1.92it/s]
2024-08-07 20:40:40,531 - INFO: validation progress:  68%|######8   | 51/75 [00:31<00:12,  1.92it/s]
2024-08-07 20:40:42,159 - INFO: validation progress:  72%|#######2  | 54/75 [00:33<00:11,  1.90it/s]
2024-08-07 20:40:43,750 - INFO: validation progress:  76%|#######6  | 57/75 [00:35<00:09,  1.90it/s]
2024-08-07 20:40:45,191 - INFO: validation progress:  80%|########  | 60/75 [00:36<00:07,  1.95it/s]
2024-08-07 20:40:46,900 - INFO: validation progress:  84%|########4 | 63/75 [00:38<00:06,  1.89it/s]
2024-08-07 20:40:48,533 - INFO: validation progress:  88%|########8 | 66/75 [00:39<00:04,  1.87it/s]
2024-08-07 20:40:50,171 - INFO: validation progress:  92%|#########2| 69/75 [00:41<00:03,  1.86it/s]
2024-08-07 20:40:51,636 - INFO: validation progress:  96%|#########6| 72/75 [00:42<00:01,  1.91it/s]
2024-08-07 20:40:53,364 - INFO: validation progress: 100%|##########| 75/75 [00:44<00:00,  1.86it/s]
2024-08-07 20:40:53,368 - INFO: validation progress: 100%|##########| 75/75 [00:44<00:00,  1.68it/s]
2024-08-07 20:40:53,666 - INFO: Validation AUC: 0.98502
2024-08-07 20:40:53,666 - INFO: Mean validation loss: 0.62995