|
2024-06-11 19:24:05,913 - INFO: Calling run.. |
|
2024-06-11 19:24:05,913 - INFO: Problem Type: text_causal_classification_modeling |
|
2024-06-11 19:24:05,913 - INFO: Global random seed: 849738 |
|
2024-06-11 19:24:05,913 - INFO: Preparing the data... |
|
2024-06-11 19:24:05,913 - INFO: Setting up automatic validation split... |
|
2024-06-11 19:24:05,955 - INFO: Preparing train and validation data |
|
2024-06-11 19:24:05,955 - INFO: Loading train dataset... |
|
2024-06-11 19:24:10,161 - INFO: Loading validation dataset... |
|
2024-06-11 19:24:10,266 - INFO: Number of observations in train dataset: 9600 |
|
2024-06-11 19:24:10,267 - INFO: Number of observations in validation dataset: 2400 |
|
2024-06-11 19:24:10,464 - WARNING: EOS token id not matching between config and tokenizer. Overwriting with tokenizer id. |
|
2024-06-11 19:24:10,465 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id. |
|
2024-06-11 19:24:10,469 - INFO: Using bfloat16 for backbone |
|
2024-06-11 19:24:10,469 - INFO: Loading tiiuae/falcon-rw-1b. This may take a while. |
|
2024-06-11 19:24:22,250 - INFO: Loaded tiiuae/falcon-rw-1b. |
|
2024-06-11 19:24:22,253 - WARNING: EOS token id not matching between generation config and tokenizer. Overwriting with tokenizer id. |
|
2024-06-11 19:24:22,253 - WARNING: PAD token id not matching between generation config and tokenizer. Overwriting with tokenizer id. |
|
2024-06-11 19:24:22,253 - INFO: Lora module names: ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h'] |
|
2024-06-11 19:24:22,374 - INFO: Enough space available for saving model weights.Required space: 2591.11MB, Available space: 995995.53MB. |
|
2024-06-11 19:24:22,379 - INFO: Optimizer AdamW has been provided with parameters {'weight_decay': 0.0, 'eps': 1e-08, 'betas': (0.8999999762, 0.9990000129), 'lr': 0.0001} |
|
2024-06-11 19:24:22,520 - WARNING: No order set for keys: ['answer_column_label', 'num_classes']. |
|
2024-06-11 19:24:22,535 - WARNING: No order set for keys: ['answer_column_label', 'num_classes']. |
|
2024-06-11 19:24:23,373 - INFO: started process: 0, can_track: True, tracking_mode: TrackingMode.DURING_EPOCH |
|
2024-06-11 19:24:23,374 - INFO: Training Epoch: 1 / 1 |
|
2024-06-11 19:24:23,374 - INFO: train loss: 0%| | 0/600 [00:00<?, ?it/s] |
|
2024-06-11 19:24:23,592 - INFO: Evaluation step: 600 |
|
2024-06-11 19:24:26,265 - INFO: train loss: 38.50: 1%|1 | 6/600 [00:02<04:46, 2.08it/s] |
|
2024-06-11 19:24:28,665 - INFO: train loss: 37.30: 2%|2 | 12/600 [00:05<04:15, 2.31it/s] |
|
2024-06-11 19:24:31,130 - INFO: train loss: 16.17: 3%|3 | 18/600 [00:07<04:06, 2.36it/s] |
|
2024-06-11 19:24:33,555 - INFO: train loss: 5.66: 4%|4 | 24/600 [00:10<03:59, 2.41it/s] |
|
2024-06-11 19:24:36,045 - INFO: train loss: 3.87: 5%|5 | 30/600 [00:12<03:56, 2.41it/s] |
|
2024-06-11 19:24:38,714 - INFO: train loss: 2.41: 6%|6 | 36/600 [00:15<03:59, 2.35it/s] |
|
2024-06-11 19:24:41,306 - INFO: train loss: 2.09: 7%|7 | 42/600 [00:17<03:58, 2.34it/s] |
|
2024-06-11 19:24:43,832 - INFO: train loss: 2.23: 8%|8 | 48/600 [00:20<03:54, 2.35it/s] |
|
2024-06-11 19:24:46,208 - INFO: train loss: 2.14: 9%|9 | 54/600 [00:22<03:47, 2.40it/s] |
|
2024-06-11 19:24:48,667 - INFO: train loss: 2.26: 10%|# | 60/600 [00:25<03:43, 2.41it/s] |
|
2024-06-11 19:24:51,185 - INFO: train loss: 1.91: 11%|#1 | 66/600 [00:27<03:42, 2.40it/s] |
|
2024-06-11 19:24:53,545 - INFO: train loss: 1.73: 12%|#2 | 72/600 [00:30<03:35, 2.44it/s] |
|
2024-06-11 19:24:55,938 - INFO: train loss: 1.85: 13%|#3 | 78/600 [00:32<03:31, 2.46it/s] |
|
2024-06-11 19:24:58,438 - INFO: train loss: 1.98: 14%|#4 | 84/600 [00:35<03:31, 2.44it/s] |
|
2024-06-11 19:25:00,935 - INFO: train loss: 1.92: 15%|#5 | 90/600 [00:37<03:29, 2.43it/s] |
|
2024-06-11 19:25:03,532 - INFO: train loss: 1.83: 16%|#6 | 96/600 [00:40<03:30, 2.39it/s] |
|
2024-06-11 19:25:06,109 - INFO: train loss: 1.60: 17%|#7 | 102/600 [00:42<03:29, 2.37it/s] |
|
2024-06-11 19:25:08,584 - INFO: train loss: 1.85: 18%|#8 | 108/600 [00:45<03:25, 2.39it/s] |
|
2024-06-11 19:25:11,083 - INFO: train loss: 2.02: 19%|#9 | 114/600 [00:47<03:23, 2.39it/s] |
|
2024-06-11 19:25:13,366 - INFO: train loss: 1.99: 20%|## | 120/600 [00:49<03:15, 2.46it/s] |
|
2024-06-11 19:25:15,909 - INFO: train loss: 1.88: 21%|##1 | 126/600 [00:52<03:15, 2.43it/s] |
|
2024-06-11 19:25:18,314 - INFO: train loss: 1.95: 22%|##2 | 132/600 [00:54<03:11, 2.45it/s] |
|
2024-06-11 19:25:20,805 - INFO: train loss: 1.82: 23%|##3 | 138/600 [00:57<03:09, 2.44it/s] |
|
2024-06-11 19:25:23,180 - INFO: train loss: 1.80: 24%|##4 | 144/600 [00:59<03:05, 2.46it/s] |
|
2024-06-11 19:25:25,788 - INFO: train loss: 1.89: 25%|##5 | 150/600 [01:02<03:06, 2.41it/s] |
|
2024-06-11 19:25:28,422 - INFO: train loss: 2.07: 26%|##6 | 156/600 [01:05<03:07, 2.37it/s] |
|
2024-06-11 19:25:30,947 - INFO: train loss: 2.02: 27%|##7 | 162/600 [01:07<03:04, 2.37it/s] |
|
2024-06-11 19:25:33,560 - INFO: train loss: 1.64: 28%|##8 | 168/600 [01:10<03:03, 2.35it/s] |
|
2024-06-11 19:25:35,986 - INFO: train loss: 1.88: 29%|##9 | 174/600 [01:12<02:58, 2.38it/s] |
|
2024-06-11 19:25:38,424 - INFO: train loss: 2.17: 30%|### | 180/600 [01:15<02:54, 2.41it/s] |
|
2024-06-11 19:25:40,841 - INFO: train loss: 1.92: 31%|###1 | 186/600 [01:17<02:50, 2.43it/s] |
|
2024-06-11 19:25:43,479 - INFO: train loss: 1.89: 32%|###2 | 192/600 [01:20<02:51, 2.38it/s] |
|
2024-06-11 19:25:46,108 - INFO: train loss: 1.71: 33%|###3 | 198/600 [01:22<02:51, 2.35it/s] |
|
2024-06-11 19:25:48,485 - INFO: train loss: 1.76: 34%|###4 | 204/600 [01:25<02:45, 2.40it/s] |
|
2024-06-11 19:25:50,947 - INFO: train loss: 1.93: 35%|###5 | 210/600 [01:27<02:41, 2.41it/s] |
|
2024-06-11 19:25:53,448 - INFO: train loss: 1.66: 36%|###6 | 216/600 [01:30<02:39, 2.41it/s] |
|
2024-06-11 19:25:55,958 - INFO: train loss: 1.70: 37%|###7 | 222/600 [01:32<02:37, 2.40it/s] |
|
2024-06-11 19:25:58,603 - INFO: train loss: 1.68: 38%|###8 | 228/600 [01:35<02:37, 2.36it/s] |
|
2024-06-11 19:26:01,255 - INFO: train loss: 1.67: 39%|###9 | 234/600 [01:37<02:37, 2.33it/s] |
|
2024-06-11 19:26:03,885 - INFO: train loss: 1.69: 40%|#### | 240/600 [01:40<02:35, 2.32it/s] |
|
2024-06-11 19:26:06,564 - INFO: train loss: 1.78: 41%|####1 | 246/600 [01:43<02:34, 2.29it/s] |
|
2024-06-11 19:26:09,095 - INFO: train loss: 1.72: 42%|####2 | 252/600 [01:45<02:30, 2.32it/s] |
|
2024-06-11 19:26:11,385 - INFO: train loss: 1.62: 43%|####3 | 258/600 [01:48<02:22, 2.40it/s] |
|
2024-06-11 19:26:13,870 - INFO: train loss: 1.52: 44%|####4 | 264/600 [01:50<02:19, 2.40it/s] |
|
2024-06-11 19:26:16,372 - INFO: train loss: 1.69: 45%|####5 | 270/600 [01:52<02:17, 2.40it/s] |
|
2024-06-11 19:26:18,692 - INFO: train loss: 1.82: 46%|####6 | 276/600 [01:55<02:12, 2.45it/s] |
|
2024-06-11 19:26:21,147 - INFO: train loss: 1.96: 47%|####6 | 282/600 [01:57<02:09, 2.45it/s] |
|
2024-06-11 19:26:23,681 - INFO: train loss: 1.83: 48%|####8 | 288/600 [02:00<02:08, 2.43it/s] |
|
2024-06-11 19:26:34,896 - INFO: train loss: 1.63: 49%|####9 | 294/600 [02:11<04:19, 1.18it/s] |
|
2024-06-11 19:26:37,499 - INFO: train loss: 1.47: 50%|##### | 300/600 [02:14<03:37, 1.38it/s] |
|
2024-06-11 19:26:40,069 - INFO: train loss: 1.69: 51%|#####1 | 306/600 [02:16<03:06, 1.57it/s] |
|
2024-06-11 19:26:42,587 - INFO: train loss: 1.60: 52%|#####2 | 312/600 [02:19<02:44, 1.75it/s] |
|
2024-06-11 19:26:44,978 - INFO: train loss: 1.44: 53%|#####3 | 318/600 [02:21<02:26, 1.93it/s] |
|
2024-06-11 19:26:47,347 - INFO: train loss: 1.38: 54%|#####4 | 324/600 [02:23<02:13, 2.08it/s] |
|
2024-06-11 19:26:49,772 - INFO: train loss: 1.22: 55%|#####5 | 330/600 [02:26<02:03, 2.18it/s] |
|
2024-06-11 19:26:52,193 - INFO: train loss: 1.34: 56%|#####6 | 336/600 [02:28<01:56, 2.26it/s] |
|
2024-06-11 19:26:54,866 - INFO: train loss: 1.70: 57%|#####6 | 342/600 [02:31<01:54, 2.26it/s] |
|
2024-06-11 19:26:57,352 - INFO: train loss: 1.63: 58%|#####8 | 348/600 [02:33<01:49, 2.30it/s] |
|
2024-06-11 19:26:59,884 - INFO: train loss: 1.51: 59%|#####8 | 354/600 [02:36<01:45, 2.32it/s] |
|
2024-06-11 19:27:02,254 - INFO: train loss: 1.32: 60%|###### | 360/600 [02:38<01:40, 2.38it/s] |
|
2024-06-11 19:27:04,630 - INFO: train loss: 1.43: 61%|######1 | 366/600 [02:41<01:36, 2.42it/s] |
|
2024-06-11 19:27:06,991 - INFO: train loss: 1.48: 62%|######2 | 372/600 [02:43<01:32, 2.46it/s] |
|
2024-06-11 19:27:09,474 - INFO: train loss: 1.45: 63%|######3 | 378/600 [02:46<01:30, 2.44it/s] |
|
2024-06-11 19:27:11,952 - INFO: train loss: 1.31: 64%|######4 | 384/600 [02:48<01:28, 2.44it/s] |
|
2024-06-11 19:27:14,439 - INFO: train loss: 1.36: 65%|######5 | 390/600 [02:51<01:26, 2.43it/s] |
|
2024-06-11 19:27:16,955 - INFO: train loss: 1.27: 66%|######6 | 396/600 [02:53<01:24, 2.42it/s] |
|
2024-06-11 19:27:19,389 - INFO: train loss: 1.36: 67%|######7 | 402/600 [02:56<01:21, 2.43it/s] |
|
2024-06-11 19:27:21,801 - INFO: train loss: 1.63: 68%|######8 | 408/600 [02:58<01:18, 2.45it/s] |
|
2024-06-11 19:27:24,365 - INFO: train loss: 1.45: 69%|######9 | 414/600 [03:00<01:17, 2.41it/s] |
|
2024-06-11 19:27:26,983 - INFO: train loss: 1.14: 70%|####### | 420/600 [03:03<01:15, 2.38it/s] |
|
2024-06-11 19:27:29,629 - INFO: train loss: 1.08: 71%|#######1 | 426/600 [03:06<01:14, 2.34it/s] |
|
2024-06-11 19:27:31,995 - INFO: train loss: 1.22: 72%|#######2 | 432/600 [03:08<01:10, 2.40it/s] |
|
2024-06-11 19:27:34,489 - INFO: train loss: 1.17: 73%|#######3 | 438/600 [03:11<01:07, 2.40it/s] |
|
2024-06-11 19:27:36,917 - INFO: train loss: 1.21: 74%|#######4 | 444/600 [03:13<01:04, 2.42it/s] |
|
2024-06-11 19:27:39,322 - INFO: train loss: 1.28: 75%|#######5 | 450/600 [03:15<01:01, 2.44it/s] |
|
2024-06-11 19:27:41,904 - INFO: train loss: 1.20: 76%|#######6 | 456/600 [03:18<00:59, 2.41it/s] |
|
2024-06-11 19:27:44,618 - INFO: train loss: 1.17: 77%|#######7 | 462/600 [03:21<00:58, 2.34it/s] |
|
2024-06-11 19:27:47,008 - INFO: train loss: 1.16: 78%|#######8 | 468/600 [03:23<00:55, 2.39it/s] |
|
2024-06-11 19:27:49,405 - INFO: train loss: 1.16: 79%|#######9 | 474/600 [03:26<00:51, 2.42it/s] |
|
2024-06-11 19:27:51,760 - INFO: train loss: 1.09: 80%|######## | 480/600 [03:28<00:48, 2.46it/s] |
|
2024-06-11 19:27:54,239 - INFO: train loss: 1.19: 81%|########1 | 486/600 [03:30<00:46, 2.45it/s] |
|
2024-06-11 19:27:56,731 - INFO: train loss: 1.25: 82%|########2 | 492/600 [03:33<00:44, 2.44it/s] |
|
2024-06-11 19:27:59,481 - INFO: train loss: 1.17: 83%|########2 | 498/600 [03:36<00:43, 2.35it/s] |
|
2024-06-11 19:28:02,025 - INFO: train loss: 1.21: 84%|########4 | 504/600 [03:38<00:40, 2.35it/s] |
|
2024-06-11 19:28:04,499 - INFO: train loss: 1.11: 85%|########5 | 510/600 [03:41<00:37, 2.38it/s] |
|
2024-06-11 19:28:07,055 - INFO: train loss: 1.09: 86%|########6 | 516/600 [03:43<00:35, 2.37it/s] |
|
2024-06-11 19:28:09,588 - INFO: train loss: 0.99: 87%|########7 | 522/600 [03:46<00:32, 2.37it/s] |
|
2024-06-11 19:28:12,009 - INFO: train loss: 1.08: 88%|########8 | 528/600 [03:48<00:30, 2.40it/s] |
|
2024-06-11 19:28:14,559 - INFO: train loss: 1.10: 89%|########9 | 534/600 [03:51<00:27, 2.39it/s] |
|
2024-06-11 19:28:17,044 - INFO: train loss: 1.03: 90%|######### | 540/600 [03:53<00:25, 2.39it/s] |
|
2024-06-11 19:28:19,569 - INFO: train loss: 1.06: 91%|#########1| 546/600 [03:56<00:22, 2.39it/s] |
|
2024-06-11 19:28:22,110 - INFO: train loss: 1.00: 92%|#########2| 552/600 [03:58<00:20, 2.38it/s] |
|
2024-06-11 19:28:24,605 - INFO: train loss: 0.95: 93%|#########3| 558/600 [04:01<00:17, 2.39it/s] |
|
2024-06-11 19:28:26,921 - INFO: train loss: 1.02: 94%|#########3| 564/600 [04:03<00:14, 2.45it/s] |
|
2024-06-11 19:28:29,323 - INFO: train loss: 1.06: 95%|#########5| 570/600 [04:05<00:12, 2.46it/s] |
|
2024-06-11 19:28:31,750 - INFO: train loss: 1.11: 96%|#########6| 576/600 [04:08<00:09, 2.46it/s] |
|
2024-06-11 19:28:34,275 - INFO: train loss: 1.13: 97%|#########7| 582/600 [04:10<00:07, 2.44it/s] |
|
2024-06-11 19:28:36,803 - INFO: train loss: 1.15: 98%|#########8| 588/600 [04:13<00:04, 2.42it/s] |
|
2024-06-11 19:28:39,449 - INFO: train loss: 1.01: 99%|#########9| 594/600 [04:16<00:02, 2.37it/s] |
|
2024-06-11 19:28:42,270 - INFO: train loss: 0.92: 100%|##########| 600/600 [04:18<00:00, 2.29it/s] |
|
2024-06-11 19:28:42,270 - INFO: train loss: 0.92: 100%|##########| 600/600 [04:18<00:00, 2.32it/s] |
|
2024-06-11 19:28:42,270 - INFO: Saving last model checkpoint to /app/output |
|
2024-06-11 19:28:42,270 - INFO: Saving checkpoint.. |
|
2024-06-11 19:28:44,868 - INFO: Starting validation inference |
|
2024-06-11 19:28:44,868 - INFO: validation progress: 0%| | 0/150 [00:00<?, ?it/s] |
|
2024-06-11 19:28:45,811 - INFO: validation progress: 5%|4 | 7/150 [00:00<00:19, 7.42it/s] |
|
2024-06-11 19:28:46,542 - INFO: validation progress: 9%|9 | 14/150 [00:01<00:15, 8.55it/s] |
|
2024-06-11 19:28:47,310 - INFO: validation progress: 14%|#4 | 21/150 [00:02<00:14, 8.80it/s] |
|
2024-06-11 19:28:48,099 - INFO: validation progress: 19%|#8 | 28/150 [00:03<00:13, 8.83it/s] |
|
2024-06-11 19:28:48,870 - INFO: validation progress: 23%|##3 | 35/150 [00:04<00:12, 8.92it/s] |
|
2024-06-11 19:28:49,672 - INFO: validation progress: 28%|##8 | 42/150 [00:04<00:12, 8.85it/s] |
|
2024-06-11 19:28:50,424 - INFO: validation progress: 33%|###2 | 49/150 [00:05<00:11, 9.00it/s] |
|
2024-06-11 19:28:51,218 - INFO: validation progress: 37%|###7 | 56/150 [00:06<00:10, 8.94it/s] |
|
2024-06-11 19:28:51,997 - INFO: validation progress: 42%|####2 | 63/150 [00:07<00:09, 8.95it/s] |
|
2024-06-11 19:28:52,814 - INFO: validation progress: 47%|####6 | 70/150 [00:07<00:09, 8.83it/s] |
|
2024-06-11 19:28:53,629 - INFO: validation progress: 51%|#####1 | 77/150 [00:08<00:08, 8.76it/s] |
|
2024-06-11 19:28:54,398 - INFO: validation progress: 56%|#####6 | 84/150 [00:09<00:07, 8.86it/s] |
|
2024-06-11 19:28:55,216 - INFO: validation progress: 61%|###### | 91/150 [00:10<00:06, 8.76it/s] |
|
2024-06-11 19:28:55,965 - INFO: validation progress: 65%|######5 | 98/150 [00:11<00:05, 8.93it/s] |
|
2024-06-11 19:28:56,706 - INFO: validation progress: 70%|####### | 105/150 [00:11<00:04, 9.08it/s] |
|
2024-06-11 19:28:57,509 - INFO: validation progress: 75%|#######4 | 112/150 [00:12<00:04, 8.97it/s] |
|
2024-06-11 19:28:58,294 - INFO: validation progress: 79%|#######9 | 119/150 [00:13<00:03, 8.95it/s] |
|
2024-06-11 19:28:59,089 - INFO: validation progress: 84%|########4 | 126/150 [00:14<00:02, 8.91it/s] |
|
2024-06-11 19:28:59,901 - INFO: validation progress: 89%|########8 | 133/150 [00:15<00:01, 8.82it/s] |
|
2024-06-11 19:29:00,664 - INFO: validation progress: 93%|#########3| 140/150 [00:15<00:01, 8.92it/s] |
|
2024-06-11 19:29:01,481 - INFO: validation progress: 98%|#########8| 147/150 [00:16<00:00, 8.81it/s] |
|
2024-06-11 19:29:01,816 - INFO: validation progress: 100%|##########| 150/150 [00:16<00:00, 8.85it/s] |
|
2024-06-11 19:29:01,876 - INFO: Validation AUC: 0.54705 |
|
2024-06-11 19:29:01,876 - INFO: Mean validation loss: 0.99504 |
|
2024-06-11 19:29:02,796 - WARNING: No order set for keys: ['answer_column_label', 'num_classes']. |
|
|
