2024-06-11 20:31:50,507 - INFO: Calling run..
2024-06-11 20:31:50,508 - INFO: Problem Type: text_causal_classification_modeling
2024-06-11 20:31:50,508 - INFO: Global random seed: 622910
2024-06-11 20:31:50,508 - INFO: Preparing the data...
2024-06-11 20:31:50,508 - INFO: Setting up automatic validation split...
2024-06-11 20:31:50,549 - INFO: Preparing train and validation data
2024-06-11 20:31:50,549 - INFO: Loading train dataset...
2024-06-11 20:31:51,325 - INFO: Loading validation dataset...
2024-06-11 20:31:51,461 - INFO: Number of observations in train dataset: 9600
2024-06-11 20:31:51,461 - INFO: Number of observations in validation dataset: 2400
2024-06-11 20:31:51,625 - WARNING: EOS token id not matching between config and tokenizer. Overwriting with tokenizer id.
2024-06-11 20:31:51,625 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id.
2024-06-11 20:31:51,632 - INFO: Using bfloat16 for backbone
2024-06-11 20:31:51,632 - INFO: Loading tiiuae/falcon-rw-1b. This may take a while.
2024-06-11 20:32:00,834 - INFO: Loaded tiiuae/falcon-rw-1b.
2024-06-11 20:32:00,836 - WARNING: EOS token id not matching between generation config and tokenizer. Overwriting with tokenizer id.
2024-06-11 20:32:00,837 - WARNING: PAD token id not matching between generation config and tokenizer. Overwriting with tokenizer id.
2024-06-11 20:32:00,837 - INFO: Lora module names: ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h']
2024-06-11 20:32:00,955 - INFO: Enough space available for saving model weights.Required space: 2591.11MB, Available space: 994803.55MB.
2024-06-11 20:32:00,959 - INFO: Optimizer AdamW has been provided with parameters {'eps': 1e-08, 'weight_decay': 0.0, 'betas': (0.8999999762, 0.9990000129), 'lr': 0.0001}
2024-06-11 20:32:01,103 - WARNING: No order set for keys: ['answer_column_label', 'num_classes'].
2024-06-11 20:32:01,123 - WARNING: No order set for keys: ['answer_column_label', 'num_classes'].
2024-06-11 20:32:02,308 - INFO: started process: 0, can_track: True, tracking_mode: TrackingMode.DURING_EPOCH
2024-06-11 20:32:02,309 - INFO: Training Epoch: 1 / 1
2024-06-11 20:32:02,310 - INFO: train loss:   0%|          | 0/600 [00:00<?, ?it/s]
2024-06-11 20:32:02,471 - INFO: Evaluation step: 600
2024-06-11 20:32:05,292 - INFO: train loss: 39.27:   1%|1         | 6/600 [00:02<04:55,  2.01it/s]
2024-06-11 20:32:07,845 - INFO: train loss: 45.45:   2%|2         | 12/600 [00:05<04:27,  2.20it/s]
2024-06-11 20:32:10,594 - INFO: train loss: 23.80:   3%|3         | 18/600 [00:08<04:25,  2.19it/s]
2024-06-11 20:32:13,096 - INFO: train loss: 5.30:   4%|4         | 24/600 [00:10<04:13,  2.27it/s]
2024-06-11 20:32:15,642 - INFO: train loss: 3.56:   5%|5         | 30/600 [00:13<04:07,  2.30it/s]
2024-06-11 20:32:18,124 - INFO: train loss: 2.98:   6%|6         | 36/600 [00:15<04:01,  2.34it/s]
2024-06-11 20:32:20,653 - INFO: train loss: 2.45:   7%|7         | 42/600 [00:18<03:57,  2.35it/s]
2024-06-11 20:32:23,248 - INFO: train loss: 2.32:   8%|8         | 48/600 [00:20<03:56,  2.34it/s]
2024-06-11 20:32:25,954 - INFO: train loss: 1.79:   9%|9         | 54/600 [00:23<03:57,  2.30it/s]
2024-06-11 20:32:28,506 - INFO: train loss: 2.08:  10%|#         | 60/600 [00:26<03:53,  2.31it/s]
2024-06-11 20:32:30,955 - INFO: train loss: 2.14:  11%|#1        | 66/600 [00:28<03:46,  2.35it/s]
2024-06-11 20:32:33,574 - INFO: train loss: 2.11:  12%|#2        | 72/600 [00:31<03:46,  2.33it/s]
2024-06-11 20:32:36,245 - INFO: train loss: 2.02:  13%|#3        | 78/600 [00:33<03:46,  2.31it/s]
2024-06-11 20:32:38,773 - INFO: train loss: 2.11:  14%|#4        | 84/600 [00:36<03:41,  2.33it/s]
2024-06-11 20:32:41,236 - INFO: train loss: 2.08:  15%|#5        | 90/600 [00:38<03:36,  2.36it/s]
2024-06-11 20:32:43,828 - INFO: train loss: 2.40:  16%|#6        | 96/600 [00:41<03:34,  2.35it/s]
2024-06-11 20:32:46,344 - INFO: train loss: 2.27:  17%|#7        | 102/600 [00:44<03:31,  2.36it/s]
2024-06-11 20:32:49,065 - INFO: train loss: 1.77:  18%|#8        | 108/600 [00:46<03:33,  2.31it/s]
2024-06-11 20:32:51,821 - INFO: train loss: 1.81:  19%|#9        | 114/600 [00:49<03:34,  2.27it/s]
2024-06-11 20:32:54,414 - INFO: train loss: 1.95:  20%|##        | 120/600 [00:52<03:30,  2.28it/s]
2024-06-11 20:32:56,900 - INFO: train loss: 1.84:  21%|##1       | 126/600 [00:54<03:24,  2.32it/s]
2024-06-11 20:32:59,366 - INFO: train loss: 1.90:  22%|##2       | 132/600 [00:57<03:18,  2.35it/s]
2024-06-11 20:33:01,981 - INFO: train loss: 1.97:  23%|##3       | 138/600 [00:59<03:17,  2.33it/s]
2024-06-11 20:33:04,669 - INFO: train loss: 1.92:  24%|##4       | 144/600 [01:02<03:18,  2.30it/s]
2024-06-11 20:33:07,232 - INFO: train loss: 1.85:  25%|##5       | 150/600 [01:04<03:14,  2.31it/s]
2024-06-11 20:33:09,831 - INFO: train loss: 1.88:  26%|##6       | 156/600 [01:07<03:11,  2.31it/s]
2024-06-11 20:33:12,242 - INFO: train loss: 1.87:  27%|##7       | 162/600 [01:09<03:05,  2.36it/s]
2024-06-11 20:33:14,922 - INFO: train loss: 1.68:  28%|##8       | 168/600 [01:12<03:05,  2.32it/s]
2024-06-11 20:33:17,402 - INFO: train loss: 1.62:  29%|##9       | 174/600 [01:15<03:01,  2.35it/s]
2024-06-11 20:33:19,889 - INFO: train loss: 1.43:  30%|###       | 180/600 [01:17<02:57,  2.37it/s]
2024-06-11 20:33:22,285 - INFO: train loss: 1.60:  31%|###1      | 186/600 [01:19<02:51,  2.41it/s]
2024-06-11 20:33:24,740 - INFO: train loss: 1.81:  32%|###2      | 192/600 [01:22<02:48,  2.42it/s]
2024-06-11 20:33:27,301 - INFO: train loss: 1.81:  33%|###3      | 198/600 [01:24<02:47,  2.40it/s]
2024-06-11 20:33:29,915 - INFO: train loss: 1.65:  34%|###4      | 204/600 [01:27<02:47,  2.36it/s]
2024-06-11 20:33:32,554 - INFO: train loss: 1.65:  35%|###5      | 210/600 [01:30<02:46,  2.34it/s]
2024-06-11 20:33:34,994 - INFO: train loss: 1.72:  36%|###6      | 216/600 [01:32<02:41,  2.37it/s]
2024-06-11 20:33:37,565 - INFO: train loss: 1.57:  37%|###7      | 222/600 [01:35<02:40,  2.36it/s]
2024-06-11 20:33:40,073 - INFO: train loss: 1.62:  38%|###8      | 228/600 [01:37<02:36,  2.37it/s]
2024-06-11 20:33:42,706 - INFO: train loss: 1.45:  39%|###9      | 234/600 [01:40<02:36,  2.34it/s]
2024-06-11 20:33:45,222 - INFO: train loss: 1.37:  40%|####      | 240/600 [01:42<02:32,  2.35it/s]
2024-06-11 20:33:47,835 - INFO: train loss: 1.46:  41%|####1     | 246/600 [01:45<02:31,  2.34it/s]
2024-06-11 20:33:50,306 - INFO: train loss: 1.42:  42%|####2     | 252/600 [01:47<02:27,  2.36it/s]
2024-06-11 20:33:52,851 - INFO: train loss: 1.46:  43%|####3     | 258/600 [01:50<02:24,  2.36it/s]
2024-06-11 20:33:55,260 - INFO: train loss: 1.64:  44%|####4     | 264/600 [01:52<02:20,  2.40it/s]
2024-06-11 20:33:57,686 - INFO: train loss: 1.86:  45%|####5     | 270/600 [01:55<02:16,  2.42it/s]
2024-06-11 20:34:00,497 - INFO: train loss: 1.53:  46%|####6     | 276/600 [01:58<02:19,  2.33it/s]
2024-06-11 20:34:02,806 - INFO: train loss: 1.37:  47%|####6     | 282/600 [02:00<02:12,  2.40it/s]
2024-06-11 20:34:05,233 - INFO: train loss: 1.35:  48%|####8     | 288/600 [02:02<02:08,  2.42it/s]
2024-06-11 20:34:07,591 - INFO: train loss: 1.42:  49%|####9     | 294/600 [02:05<02:04,  2.46it/s]
2024-06-11 20:34:10,121 - INFO: train loss: 1.36:  50%|#####     | 300/600 [02:07<02:03,  2.43it/s]
2024-06-11 20:34:12,788 - INFO: train loss: 1.24:  51%|#####1    | 306/600 [02:10<02:03,  2.37it/s]
2024-06-11 20:34:15,105 - INFO: train loss: 1.01:  52%|#####2    | 312/600 [02:12<01:58,  2.43it/s]
2024-06-11 20:34:17,567 - INFO: train loss: 1.19:  53%|#####3    | 318/600 [02:15<01:55,  2.44it/s]
2024-06-11 20:34:19,920 - INFO: train loss: 1.22:  54%|#####4    | 324/600 [02:17<01:51,  2.47it/s]
2024-06-11 20:34:22,339 - INFO: train loss: 1.26:  55%|#####5    | 330/600 [02:20<01:49,  2.47it/s]
2024-06-11 20:34:24,935 - INFO: train loss: 1.11:  56%|#####6    | 336/600 [02:22<01:49,  2.42it/s]
2024-06-11 20:34:27,728 - INFO: train loss: 1.11:  57%|#####6    | 342/600 [02:25<01:50,  2.33it/s]
2024-06-11 20:34:30,159 - INFO: train loss: 1.08:  58%|#####8    | 348/600 [02:27<01:46,  2.37it/s]
2024-06-11 20:34:32,641 - INFO: train loss: 0.94:  59%|#####8    | 354/600 [02:30<01:43,  2.39it/s]
2024-06-11 20:34:35,057 - INFO: train loss: 1.11:  60%|######    | 360/600 [02:32<01:39,  2.41it/s]
2024-06-11 20:34:37,412 - INFO: train loss: 1.18:  61%|######1   | 366/600 [02:35<01:35,  2.45it/s]
2024-06-11 20:34:39,890 - INFO: train loss: 1.03:  62%|######2   | 372/600 [02:37<01:33,  2.44it/s]
2024-06-11 20:34:42,275 - INFO: train loss: 1.00:  63%|######3   | 378/600 [02:39<01:30,  2.46it/s]
2024-06-11 20:34:44,903 - INFO: train loss: 1.02:  64%|######4   | 384/600 [02:42<01:29,  2.41it/s]
2024-06-11 20:34:47,349 - INFO: train loss: 0.91:  65%|######5   | 390/600 [02:45<01:26,  2.42it/s]
2024-06-11 20:34:49,839 - INFO: train loss: 0.72:  66%|######6   | 396/600 [02:47<01:24,  2.42it/s]
2024-06-11 20:34:52,489 - INFO: train loss: 0.81:  67%|######7   | 402/600 [02:50<01:23,  2.37it/s]
2024-06-11 20:34:55,035 - INFO: train loss: 0.77:  68%|######8   | 408/600 [02:52<01:21,  2.37it/s]
2024-06-11 20:34:57,554 - INFO: train loss: 0.80:  69%|######9   | 414/600 [02:55<01:18,  2.37it/s]
2024-06-11 20:35:00,243 - INFO: train loss: 0.62:  70%|#######   | 420/600 [02:57<01:17,  2.33it/s]
2024-06-11 20:35:02,714 - INFO: train loss: 0.72:  71%|#######1  | 426/600 [03:00<01:13,  2.36it/s]
2024-06-11 20:35:05,239 - INFO: train loss: 0.80:  72%|#######2  | 432/600 [03:02<01:11,  2.36it/s]
2024-06-11 20:35:07,446 - INFO: train loss: 0.65:  73%|#######3  | 438/600 [03:05<01:05,  2.46it/s]
2024-06-11 20:35:09,916 - INFO: train loss: 0.51:  74%|#######4  | 444/600 [03:07<01:03,  2.45it/s]
2024-06-11 20:35:12,415 - INFO: train loss: 0.43:  75%|#######5  | 450/600 [03:10<01:01,  2.43it/s]
2024-06-11 20:35:14,980 - INFO: train loss: 0.50:  76%|#######6  | 456/600 [03:12<00:59,  2.41it/s]
2024-06-11 20:35:17,754 - INFO: train loss: 0.56:  77%|#######7  | 462/600 [03:15<00:59,  2.33it/s]
2024-06-11 20:35:20,220 - INFO: train loss: 0.67:  78%|#######8  | 468/600 [03:17<00:55,  2.36it/s]
2024-06-11 20:35:22,599 - INFO: train loss: 0.53:  79%|#######9  | 474/600 [03:20<00:52,  2.40it/s]
2024-06-11 20:35:25,068 - INFO: train loss: 0.56:  80%|########  | 480/600 [03:22<00:49,  2.41it/s]
2024-06-11 20:35:27,661 - INFO: train loss: 0.68:  81%|########1 | 486/600 [03:25<00:47,  2.38it/s]
2024-06-11 20:35:30,067 - INFO: train loss: 0.70:  82%|########2 | 492/600 [03:27<00:44,  2.41it/s]
2024-06-11 20:35:32,686 - INFO: train loss: 0.49:  83%|########2 | 498/600 [03:30<00:42,  2.38it/s]
2024-06-11 20:35:35,154 - INFO: train loss: 0.51:  84%|########4 | 504/600 [03:32<00:40,  2.39it/s]
2024-06-11 20:35:37,517 - INFO: train loss: 0.53:  85%|########5 | 510/600 [03:35<00:36,  2.43it/s]
2024-06-11 20:35:39,862 - INFO: train loss: 0.48:  86%|########6 | 516/600 [03:37<00:34,  2.47it/s]
2024-06-11 20:35:42,367 - INFO: train loss: 0.56:  87%|########7 | 522/600 [03:40<00:31,  2.45it/s]
2024-06-11 20:35:44,941 - INFO: train loss: 0.65:  88%|########8 | 528/600 [03:42<00:29,  2.41it/s]
2024-06-11 20:35:47,450 - INFO: train loss: 0.62:  89%|########9 | 534/600 [03:45<00:27,  2.41it/s]
2024-06-11 20:35:50,018 - INFO: train loss: 0.73:  90%|######### | 540/600 [03:47<00:25,  2.38it/s]
2024-06-11 20:35:52,437 - INFO: train loss: 0.63:  91%|#########1| 546/600 [03:50<00:22,  2.41it/s]
2024-06-11 20:35:54,985 - INFO: train loss: 0.55:  92%|#########2| 552/600 [03:52<00:20,  2.39it/s]
2024-06-11 20:35:57,432 - INFO: train loss: 0.59:  93%|#########3| 558/600 [03:55<00:17,  2.41it/s]
2024-06-11 20:35:59,896 - INFO: train loss: 0.54:  94%|#########3| 564/600 [03:57<00:14,  2.42it/s]
2024-06-11 20:36:02,380 - INFO: train loss: 0.60:  95%|#########5| 570/600 [04:00<00:12,  2.42it/s]
2024-06-11 20:36:04,784 - INFO: train loss: 0.64:  96%|#########6| 576/600 [04:02<00:09,  2.44it/s]
2024-06-11 20:36:07,386 - INFO: train loss: 0.68:  97%|#########7| 582/600 [04:05<00:07,  2.40it/s]
2024-06-11 20:36:09,709 - INFO: train loss: 0.64:  98%|#########8| 588/600 [04:07<00:04,  2.45it/s]
2024-06-11 20:36:12,267 - INFO: train loss: 0.56:  99%|#########9| 594/600 [04:09<00:02,  2.42it/s]
2024-06-11 20:36:14,683 - INFO: train loss: 0.51: 100%|##########| 600/600 [04:12<00:00,  2.44it/s]
2024-06-11 20:36:14,683 - INFO: train loss: 0.51: 100%|##########| 600/600 [04:12<00:00,  2.38it/s]
2024-06-11 20:36:14,683 - INFO: Saving last model checkpoint to /app/output
2024-06-11 20:36:14,683 - INFO: Saving checkpoint..
2024-06-11 20:36:17,205 - INFO: Starting validation inference
2024-06-11 20:36:17,205 - INFO: validation progress:   0%|          | 0/150 [00:00<?, ?it/s]
2024-06-11 20:36:18,147 - INFO: validation progress:   5%|4         | 7/150 [00:00<00:19,  7.43it/s]
2024-06-11 20:36:18,877 - INFO: validation progress:   9%|9         | 14/150 [00:01<00:15,  8.57it/s]
2024-06-11 20:36:19,646 - INFO: validation progress:  14%|#4        | 21/150 [00:02<00:14,  8.80it/s]
2024-06-11 20:36:20,434 - INFO: validation progress:  19%|#8        | 28/150 [00:03<00:13,  8.83it/s]
2024-06-11 20:36:21,206 - INFO: validation progress:  23%|##3       | 35/150 [00:04<00:12,  8.92it/s]
2024-06-11 20:36:22,012 - INFO: validation progress:  28%|##8       | 42/150 [00:04<00:12,  8.84it/s]
2024-06-11 20:36:22,764 - INFO: validation progress:  33%|###2      | 49/150 [00:05<00:11,  8.98it/s]
2024-06-11 20:36:23,541 - INFO: validation progress:  37%|###7      | 56/150 [00:06<00:10,  8.99it/s]
2024-06-11 20:36:24,317 - INFO: validation progress:  42%|####2     | 63/150 [00:07<00:09,  9.00it/s]
2024-06-11 20:36:25,136 - INFO: validation progress:  47%|####6     | 70/150 [00:07<00:09,  8.86it/s]
2024-06-11 20:36:25,949 - INFO: validation progress:  51%|#####1    | 77/150 [00:08<00:08,  8.78it/s]
2024-06-11 20:36:26,719 - INFO: validation progress:  56%|#####6    | 84/150 [00:09<00:07,  8.87it/s]
2024-06-11 20:36:27,539 - INFO: validation progress:  61%|######    | 91/150 [00:10<00:06,  8.77it/s]
2024-06-11 20:36:28,286 - INFO: validation progress:  65%|######5   | 98/150 [00:11<00:05,  8.94it/s]
2024-06-11 20:36:29,026 - INFO: validation progress:  70%|#######   | 105/150 [00:11<00:04,  9.09it/s]
2024-06-11 20:36:29,827 - INFO: validation progress:  75%|#######4  | 112/150 [00:12<00:04,  8.98it/s]
2024-06-11 20:36:30,617 - INFO: validation progress:  79%|#######9  | 119/150 [00:13<00:03,  8.95it/s]
2024-06-11 20:36:31,410 - INFO: validation progress:  84%|########4 | 126/150 [00:14<00:02,  8.91it/s]
2024-06-11 20:36:32,219 - INFO: validation progress:  89%|########8 | 133/150 [00:15<00:01,  8.83it/s]
2024-06-11 20:36:32,983 - INFO: validation progress:  93%|#########3| 140/150 [00:15<00:01,  8.93it/s]
2024-06-11 20:36:33,778 - INFO: validation progress:  98%|#########8| 147/150 [00:16<00:00,  8.89it/s]
2024-06-11 20:36:34,114 - INFO: validation progress: 100%|##########| 150/150 [00:16<00:00,  8.87it/s]
2024-06-11 20:36:34,172 - INFO: Validation AUC: 0.75388
2024-06-11 20:36:34,173 - INFO: Mean validation loss: 0.51912
2024-06-11 20:36:35,118 - WARNING: No order set for keys: ['answer_column_label', 'num_classes'].