|
2024-06-11 18:12:02,678 - INFO: Calling run.. |
|
2024-06-11 18:12:02,678 - INFO: Problem Type: text_causal_classification_modeling |
|
2024-06-11 18:12:02,678 - INFO: Global random seed: 5001 |
|
2024-06-11 18:12:02,678 - INFO: Preparing the data... |
|
2024-06-11 18:12:02,679 - INFO: Setting up automatic validation split... |
|
2024-06-11 18:12:02,720 - INFO: Preparing train and validation data |
|
2024-06-11 18:12:02,721 - INFO: Loading train dataset... |
|
2024-06-11 18:12:03,290 - INFO: Loading validation dataset... |
|
2024-06-11 18:12:03,398 - INFO: Number of observations in train dataset: 9600 |
|
2024-06-11 18:12:03,399 - INFO: Number of observations in validation dataset: 2400 |
|
2024-06-11 18:12:03,566 - WARNING: EOS token id not matching between config and tokenizer. Overwriting with tokenizer id. |
|
2024-06-11 18:12:03,566 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id. |
|
2024-06-11 18:12:03,571 - INFO: Using bfloat16 for backbone |
|
2024-06-11 18:12:03,571 - INFO: Loading tiiuae/falcon-rw-1b. This may take a while. |
|
2024-06-11 18:13:13,609 - INFO: Loaded tiiuae/falcon-rw-1b. |
|
2024-06-11 18:13:13,612 - WARNING: EOS token id not matching between generation config and tokenizer. Overwriting with tokenizer id. |
|
2024-06-11 18:13:13,612 - WARNING: PAD token id not matching between generation config and tokenizer. Overwriting with tokenizer id. |
|
2024-06-11 18:13:13,612 - INFO: Lora module names: ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h'] |
|
2024-06-11 18:13:13,747 - INFO: Enough space available for saving model weights.Required space: 2591.11MB, Available space: 996275.30MB. |
|
2024-06-11 18:13:13,752 - INFO: Optimizer AdamW has been provided with parameters {'weight_decay': 0.0, 'eps': 1e-08, 'betas': (0.8999999762, 0.9990000129), 'lr': 0.0001} |
|
2024-06-11 18:13:13,897 - WARNING: No order set for keys: ['answer_column_label', 'num_classes']. |
|
2024-06-11 18:13:13,912 - WARNING: No order set for keys: ['answer_column_label', 'num_classes']. |
|
2024-06-11 18:13:14,944 - INFO: started process: 0, can_track: True, tracking_mode: TrackingMode.DURING_EPOCH |
|
2024-06-11 18:13:14,945 - INFO: Training Epoch: 1 / 1 |
|
2024-06-11 18:13:14,945 - INFO: train loss: 0%| | 0/600 [00:00<?, ?it/s] |
|
2024-06-11 18:13:15,125 - INFO: Evaluation step: 600 |
|
2024-06-11 18:13:17,794 - INFO: train loss: 27.85: 1%|1 | 6/600 [00:02<04:41, 2.11it/s] |
|
2024-06-11 18:13:20,240 - INFO: train loss: 29.97: 2%|2 | 12/600 [00:05<04:15, 2.30it/s] |
|
2024-06-11 18:13:22,856 - INFO: train loss: 15.00: 3%|3 | 18/600 [00:07<04:13, 2.30it/s] |
|
2024-06-11 18:13:25,363 - INFO: train loss: 4.29: 4%|4 | 24/600 [00:10<04:06, 2.33it/s] |
|
2024-06-11 18:13:27,809 - INFO: train loss: 2.71: 5%|5 | 30/600 [00:12<04:00, 2.37it/s] |
|
2024-06-11 18:13:30,444 - INFO: train loss: 2.08: 6%|6 | 36/600 [00:15<04:00, 2.34it/s] |
|
2024-06-11 18:13:32,938 - INFO: train loss: 2.43: 7%|7 | 42/600 [00:17<03:56, 2.36it/s] |
|
2024-06-11 18:13:35,519 - INFO: train loss: 2.30: 8%|8 | 48/600 [00:20<03:54, 2.35it/s] |
|
2024-06-11 18:13:37,862 - INFO: train loss: 2.07: 9%|9 | 54/600 [00:22<03:46, 2.41it/s] |
|
2024-06-11 18:13:40,492 - INFO: train loss: 2.11: 10%|# | 60/600 [00:25<03:47, 2.37it/s] |
|
2024-06-11 18:13:43,074 - INFO: train loss: 2.00: 11%|#1 | 66/600 [00:28<03:46, 2.36it/s] |
|
2024-06-11 18:13:46,074 - INFO: train loss: 1.87: 12%|#2 | 72/600 [00:31<03:56, 2.23it/s] |
|
2024-06-11 18:13:48,617 - INFO: train loss: 1.80: 13%|#3 | 78/600 [00:33<03:49, 2.27it/s] |
|
2024-06-11 18:13:51,159 - INFO: train loss: 1.96: 14%|#4 | 84/600 [00:36<03:44, 2.30it/s] |
|
2024-06-11 18:13:53,541 - INFO: train loss: 3.41: 15%|#5 | 90/600 [00:38<03:36, 2.36it/s] |
|
2024-06-11 18:13:56,135 - INFO: train loss: 3.75: 16%|#6 | 96/600 [00:41<03:34, 2.35it/s] |
|
2024-06-11 18:13:58,539 - INFO: train loss: 2.11: 17%|#7 | 102/600 [00:43<03:28, 2.39it/s] |
|
2024-06-11 18:14:01,301 - INFO: train loss: 1.87: 18%|#8 | 108/600 [00:46<03:32, 2.32it/s] |
|
2024-06-11 18:14:03,837 - INFO: train loss: 1.84: 19%|#9 | 114/600 [00:48<03:28, 2.33it/s] |
|
2024-06-11 18:14:06,304 - INFO: train loss: 1.74: 20%|## | 120/600 [00:51<03:23, 2.36it/s] |
|
2024-06-11 18:14:08,749 - INFO: train loss: 1.78: 21%|##1 | 126/600 [00:53<03:18, 2.39it/s] |
|
2024-06-11 18:14:11,182 - INFO: train loss: 1.63: 22%|##2 | 132/600 [00:56<03:14, 2.41it/s] |
|
2024-06-11 18:14:13,724 - INFO: train loss: 1.79: 23%|##3 | 138/600 [00:58<03:12, 2.40it/s] |
|
2024-06-11 18:14:16,072 - INFO: train loss: 2.02: 24%|##4 | 144/600 [01:01<03:06, 2.44it/s] |
|
2024-06-11 18:14:18,388 - INFO: train loss: 1.65: 25%|##5 | 150/600 [01:03<03:01, 2.48it/s] |
|
2024-06-11 18:14:20,832 - INFO: train loss: 1.57: 26%|##6 | 156/600 [01:05<02:59, 2.48it/s] |
|
2024-06-11 18:14:23,260 - INFO: train loss: 1.52: 27%|##7 | 162/600 [01:08<02:57, 2.47it/s] |
|
2024-06-11 18:14:26,004 - INFO: train loss: 1.90: 28%|##8 | 168/600 [01:11<03:01, 2.38it/s] |
|
2024-06-11 18:14:28,649 - INFO: train loss: 1.86: 29%|##9 | 174/600 [01:13<03:01, 2.35it/s] |
|
2024-06-11 18:14:31,091 - INFO: train loss: 1.83: 30%|### | 180/600 [01:16<02:56, 2.38it/s] |
|
2024-06-11 18:14:33,524 - INFO: train loss: 1.74: 31%|###1 | 186/600 [01:18<02:52, 2.40it/s] |
|
2024-06-11 18:14:36,108 - INFO: train loss: 1.61: 32%|###2 | 192/600 [01:21<02:51, 2.38it/s] |
|
2024-06-11 18:14:38,735 - INFO: train loss: 1.75: 33%|###3 | 198/600 [01:23<02:51, 2.35it/s] |
|
2024-06-11 18:14:41,312 - INFO: train loss: 1.74: 34%|###4 | 204/600 [01:26<02:49, 2.34it/s] |
|
2024-06-11 18:14:43,836 - INFO: train loss: 1.58: 35%|###5 | 210/600 [01:28<02:45, 2.35it/s] |
|
2024-06-11 18:14:46,349 - INFO: train loss: 1.73: 36%|###6 | 216/600 [01:31<02:42, 2.36it/s] |
|
2024-06-11 18:14:48,860 - INFO: train loss: 1.72: 37%|###7 | 222/600 [01:33<02:39, 2.37it/s] |
|
2024-06-11 18:14:51,344 - INFO: train loss: 1.50: 38%|###8 | 228/600 [01:36<02:36, 2.38it/s] |
|
2024-06-11 18:14:53,917 - INFO: train loss: 1.57: 39%|###9 | 234/600 [01:38<02:34, 2.37it/s] |
|
2024-06-11 18:14:56,442 - INFO: train loss: 1.57: 40%|#### | 240/600 [01:41<02:31, 2.37it/s] |
|
2024-06-11 18:14:58,801 - INFO: train loss: 1.45: 41%|####1 | 246/600 [01:43<02:26, 2.42it/s] |
|
2024-06-11 18:15:01,446 - INFO: train loss: 1.75: 42%|####2 | 252/600 [01:46<02:26, 2.37it/s] |
|
2024-06-11 18:15:03,751 - INFO: train loss: 1.38: 43%|####3 | 258/600 [01:48<02:20, 2.44it/s] |
|
2024-06-11 18:15:06,382 - INFO: train loss: 1.25: 44%|####4 | 264/600 [01:51<02:20, 2.39it/s] |
|
2024-06-11 18:15:09,001 - INFO: train loss: 1.29: 45%|####5 | 270/600 [01:54<02:19, 2.36it/s] |
|
2024-06-11 18:15:11,157 - INFO: train loss: 1.34: 46%|####6 | 276/600 [01:56<02:11, 2.47it/s] |
|
2024-06-11 18:15:13,775 - INFO: train loss: 1.53: 47%|####6 | 282/600 [01:58<02:11, 2.41it/s] |
|
2024-06-11 18:15:16,367 - INFO: train loss: 1.46: 48%|####8 | 288/600 [02:01<02:10, 2.38it/s] |
|
2024-06-11 18:15:18,800 - INFO: train loss: 1.29: 49%|####9 | 294/600 [02:03<02:07, 2.41it/s] |
|
2024-06-11 18:15:21,149 - INFO: train loss: 1.11: 50%|##### | 300/600 [02:06<02:02, 2.45it/s] |
|
2024-06-11 18:15:23,366 - INFO: train loss: 1.10: 51%|#####1 | 306/600 [02:08<01:56, 2.52it/s] |
|
2024-06-11 18:15:25,956 - INFO: train loss: 1.38: 52%|#####2 | 312/600 [02:11<01:57, 2.46it/s] |
|
2024-06-11 18:15:28,552 - INFO: train loss: 1.22: 53%|#####3 | 318/600 [02:13<01:56, 2.41it/s] |
|
2024-06-11 18:15:31,098 - INFO: train loss: 1.36: 54%|#####4 | 324/600 [02:16<01:55, 2.39it/s] |
|
2024-06-11 18:15:33,640 - INFO: train loss: 1.26: 55%|#####5 | 330/600 [02:18<01:53, 2.38it/s] |
|
2024-06-11 18:15:36,165 - INFO: train loss: 1.25: 56%|#####6 | 336/600 [02:21<01:50, 2.38it/s] |
|
2024-06-11 18:15:38,498 - INFO: train loss: 1.21: 57%|#####6 | 342/600 [02:23<01:45, 2.44it/s] |
|
2024-06-11 18:15:40,974 - INFO: train loss: 1.17: 58%|#####8 | 348/600 [02:26<01:43, 2.43it/s] |
|
2024-06-11 18:15:43,399 - INFO: train loss: 0.98: 59%|#####8 | 354/600 [02:28<01:40, 2.44it/s] |
|
2024-06-11 18:15:45,959 - INFO: train loss: 0.75: 60%|###### | 360/600 [02:31<01:39, 2.41it/s] |
|
2024-06-11 18:15:48,574 - INFO: train loss: 0.90: 61%|######1 | 366/600 [02:33<01:38, 2.38it/s] |
|
2024-06-11 18:15:50,821 - INFO: train loss: 0.93: 62%|######2 | 372/600 [02:35<01:32, 2.46it/s] |
|
2024-06-11 18:15:53,266 - INFO: train loss: 0.92: 63%|######3 | 378/600 [02:38<01:30, 2.46it/s] |
|
2024-06-11 18:15:55,825 - INFO: train loss: 0.87: 64%|######4 | 384/600 [02:40<01:29, 2.42it/s] |
|
2024-06-11 18:15:58,587 - INFO: train loss: 0.72: 65%|######5 | 390/600 [02:43<01:29, 2.34it/s] |
|
2024-06-11 18:16:01,176 - INFO: train loss: 0.80: 66%|######6 | 396/600 [02:46<01:27, 2.33it/s] |
|
2024-06-11 18:16:03,717 - INFO: train loss: 0.89: 67%|######7 | 402/600 [02:48<01:24, 2.34it/s] |
|
2024-06-11 18:16:06,313 - INFO: train loss: 0.97: 68%|######8 | 408/600 [02:51<01:22, 2.33it/s] |
|
2024-06-11 18:16:08,801 - INFO: train loss: 0.79: 69%|######9 | 414/600 [02:53<01:18, 2.36it/s] |
|
2024-06-11 18:16:11,184 - INFO: train loss: 0.76: 70%|####### | 420/600 [02:56<01:14, 2.40it/s] |
|
2024-06-11 18:16:13,644 - INFO: train loss: 0.76: 71%|#######1 | 426/600 [02:58<01:12, 2.41it/s] |
|
2024-06-11 18:16:16,306 - INFO: train loss: 0.64: 72%|#######2 | 432/600 [03:01<01:11, 2.36it/s] |
|
2024-06-11 18:16:18,760 - INFO: train loss: 0.61: 73%|#######3 | 438/600 [03:03<01:07, 2.39it/s] |
|
2024-06-11 18:16:21,223 - INFO: train loss: 0.59: 74%|#######4 | 444/600 [03:06<01:04, 2.40it/s] |
|
2024-06-11 18:16:23,589 - INFO: train loss: 0.72: 75%|#######5 | 450/600 [03:08<01:01, 2.44it/s] |
|
2024-06-11 18:16:26,060 - INFO: train loss: 0.59: 76%|#######6 | 456/600 [03:11<00:59, 2.44it/s] |
|
2024-06-11 18:16:28,667 - INFO: train loss: 0.70: 77%|#######7 | 462/600 [03:13<00:57, 2.39it/s] |
|
2024-06-11 18:16:31,109 - INFO: train loss: 0.69: 78%|#######8 | 468/600 [03:16<00:54, 2.41it/s] |
|
2024-06-11 18:16:33,545 - INFO: train loss: 0.61: 79%|#######9 | 474/600 [03:18<00:51, 2.43it/s] |
|
2024-06-11 18:16:35,943 - INFO: train loss: 0.42: 80%|######## | 480/600 [03:20<00:48, 2.45it/s] |
|
2024-06-11 18:16:38,289 - INFO: train loss: 0.57: 81%|########1 | 486/600 [03:23<00:45, 2.48it/s] |
|
2024-06-11 18:16:40,909 - INFO: train loss: 0.63: 82%|########2 | 492/600 [03:25<00:44, 2.42it/s] |
|
2024-06-11 18:16:43,398 - INFO: train loss: 0.54: 83%|########2 | 498/600 [03:28<00:42, 2.42it/s] |
|
2024-06-11 18:16:45,881 - INFO: train loss: 0.58: 84%|########4 | 504/600 [03:30<00:39, 2.42it/s] |
|
2024-06-11 18:16:48,275 - INFO: train loss: 0.63: 85%|########5 | 510/600 [03:33<00:36, 2.44it/s] |
|
2024-06-11 18:16:50,832 - INFO: train loss: 0.57: 86%|########6 | 516/600 [03:35<00:34, 2.41it/s] |
|
2024-06-11 18:16:53,394 - INFO: train loss: 0.59: 87%|########7 | 522/600 [03:38<00:32, 2.39it/s] |
|
2024-06-11 18:16:55,948 - INFO: train loss: 0.59: 88%|########8 | 528/600 [03:41<00:30, 2.38it/s] |
|
2024-06-11 18:16:58,552 - INFO: train loss: 0.71: 89%|########9 | 534/600 [03:43<00:28, 2.36it/s] |
|
2024-06-11 18:17:01,223 - INFO: train loss: 0.63: 90%|######### | 540/600 [03:46<00:25, 2.32it/s] |
|
2024-06-11 18:17:03,727 - INFO: train loss: 0.65: 91%|#########1| 546/600 [03:48<00:23, 2.34it/s] |
|
2024-06-11 18:17:06,060 - INFO: train loss: 0.63: 92%|#########2| 552/600 [03:51<00:19, 2.41it/s] |
|
2024-06-11 18:17:08,627 - INFO: train loss: 0.65: 93%|#########3| 558/600 [03:53<00:17, 2.39it/s] |
|
2024-06-11 18:17:11,043 - INFO: train loss: 0.62: 94%|#########3| 564/600 [03:56<00:14, 2.41it/s] |
|
2024-06-11 18:17:13,728 - INFO: train loss: 0.64: 95%|#########5| 570/600 [03:58<00:12, 2.36it/s] |
|
2024-06-11 18:17:16,137 - INFO: train loss: 0.62: 96%|#########6| 576/600 [04:01<00:10, 2.40it/s] |
|
2024-06-11 18:17:18,768 - INFO: train loss: 0.72: 97%|#########7| 582/600 [04:03<00:07, 2.36it/s] |
|
2024-06-11 18:17:21,126 - INFO: train loss: 0.79: 98%|#########8| 588/600 [04:06<00:04, 2.41it/s] |
|
2024-06-11 18:17:23,619 - INFO: train loss: 0.76: 99%|#########9| 594/600 [04:08<00:02, 2.41it/s] |
|
2024-06-11 18:17:25,921 - INFO: train loss: 0.61: 100%|##########| 600/600 [04:10<00:00, 2.47it/s] |
|
2024-06-11 18:17:25,921 - INFO: train loss: 0.61: 100%|##########| 600/600 [04:10<00:00, 2.39it/s] |
|
2024-06-11 18:17:25,921 - INFO: Saving last model checkpoint to /app/output |
|
2024-06-11 18:17:25,922 - INFO: Saving checkpoint.. |
|
2024-06-11 18:17:28,646 - INFO: Starting validation inference |
|
2024-06-11 18:17:28,646 - INFO: validation progress: 0%| | 0/150 [00:00<?, ?it/s] |
|
2024-06-11 18:17:29,623 - INFO: validation progress: 5%|4 | 7/150 [00:00<00:19, 7.17it/s] |
|
2024-06-11 18:17:30,353 - INFO: validation progress: 9%|9 | 14/150 [00:01<00:16, 8.42it/s] |
|
2024-06-11 18:17:31,126 - INFO: validation progress: 14%|#4 | 21/150 [00:02<00:14, 8.70it/s] |
|
2024-06-11 18:17:31,912 - INFO: validation progress: 19%|#8 | 28/150 [00:03<00:13, 8.78it/s] |
|
2024-06-11 18:17:32,691 - INFO: validation progress: 23%|##3 | 35/150 [00:04<00:12, 8.85it/s] |
|
2024-06-11 18:17:33,491 - INFO: validation progress: 28%|##8 | 42/150 [00:04<00:12, 8.82it/s] |
|
2024-06-11 18:17:34,243 - INFO: validation progress: 33%|###2 | 49/150 [00:05<00:11, 8.97it/s] |
|
2024-06-11 18:17:35,022 - INFO: validation progress: 37%|###7 | 56/150 [00:06<00:10, 8.98it/s] |
|
2024-06-11 18:17:35,800 - INFO: validation progress: 42%|####2 | 63/150 [00:07<00:09, 8.98it/s] |
|
2024-06-11 18:17:36,616 - INFO: validation progress: 47%|####6 | 70/150 [00:07<00:09, 8.85it/s] |
|
2024-06-11 18:17:37,430 - INFO: validation progress: 51%|#####1 | 77/150 [00:08<00:08, 8.77it/s] |
|
2024-06-11 18:17:38,207 - INFO: validation progress: 56%|#####6 | 84/150 [00:09<00:07, 8.85it/s] |
|
2024-06-11 18:17:39,024 - INFO: validation progress: 61%|###### | 91/150 [00:10<00:06, 8.76it/s] |
|
2024-06-11 18:17:39,772 - INFO: validation progress: 65%|######5 | 98/150 [00:11<00:05, 8.93it/s] |
|
2024-06-11 18:17:40,514 - INFO: validation progress: 70%|####### | 105/150 [00:11<00:04, 9.08it/s] |
|
2024-06-11 18:17:41,317 - INFO: validation progress: 75%|#######4 | 112/150 [00:12<00:04, 8.97it/s] |
|
2024-06-11 18:17:42,101 - INFO: validation progress: 79%|#######9 | 119/150 [00:13<00:03, 8.96it/s] |
|
2024-06-11 18:17:42,894 - INFO: validation progress: 84%|########4 | 126/150 [00:14<00:02, 8.92it/s] |
|
2024-06-11 18:17:43,703 - INFO: validation progress: 89%|########8 | 133/150 [00:15<00:01, 8.83it/s] |
|
2024-06-11 18:17:44,475 - INFO: validation progress: 93%|#########3| 140/150 [00:15<00:01, 8.90it/s] |
|
2024-06-11 18:17:45,272 - INFO: validation progress: 98%|#########8| 147/150 [00:16<00:00, 8.87it/s] |
|
2024-06-11 18:17:45,607 - INFO: validation progress: 100%|##########| 150/150 [00:16<00:00, 8.84it/s] |
|
2024-06-11 18:17:45,678 - INFO: Validation AUC: 0.65136 |
|
2024-06-11 18:17:45,679 - INFO: Mean validation loss: 0.53393 |
|
2024-06-11 18:17:46,646 - WARNING: No order set for keys: ['answer_column_label', 'num_classes']. |
|
|
