Starting training with configuration: ====================================== Error log: logs/error_20250401_104945.log PYTHONPATH: :/home/deeptanshul/Toxic-Comment-Classification-using-Deep-Learning ====================================== Starting training in background... Training process started with PID: 7731 Monitor commands: 1. View error log: tail -f logs/error_20250401_104945.log 2. Check process status: ps -p 7731 3. Stop training: kill 7731 Warning: TF32 not supported on this GPU. Disabling. Error during training: Missing label columns in dataset: ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate'] Performing cleanup... Fatal error: Missing label columns in dataset: ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate'] Performing cleanup... Performing cleanup... Warning: TF32 not supported on this GPU. Disabling. Initialized dataset with 285264 samples Loading sample 147000 Loading sample 225000 Loading sample 4000 Loading sample 86000 Loading sample 50000 Loading sample 144000 Loading sample 42000 Loading sample 244000 Loading sample 229000 Loading sample 210000 Loading sample 116000 Loading sample 278000 Loading sample 154000 Loading sample 227000 Loading sample 272000 Loading sample 224000 Loading sample 237000 Loading sample 77000 Loading sample 134000 Loading sample 201000 Loading sample 211000 Loading sample 65000 Loading sample 231000 Loading sample 194000 Loading sample 200000 Loading sample 153000 Loading sample 211000 Loading sample 195000 Loading sample 59000 Loading sample 134000 Loading sample 5000 Loading sample 264000 Loading sample 9000 Loading sample 273000 Loading sample 114000 Loading sample 20000 Loading sample 240000 Loading sample 39000 Loading sample 195000 Loading sample 263000 Loading sample 265000 Loading sample 119000 Loading sample 15000 Loading sample 30000 Loading sample 141000 Loading sample 28000 Loading sample 94000 Loading sample 157000 Loading sample 185000 Loading sample 227000 Loading sample 132000 Loading sample 152000 Loading sample 15000 Loading sample 192000 Loading sample 211000 Loading sample 173000 Loading sample 67000 Loading sample 200000 Loading sample 52000 Loading sample 280000 Loading sample 0 Loading sample 157000 Loading sample 72000 Loading sample 278000 Loading sample 198000 Loading sample 179000 Loading sample 27000 Loading sample 33000 Loading sample 221000 Loading sample 231000 Loading sample 144000 Loading sample 235000 Loading sample 42000 Loading sample 155000 Loading sample 155000 Loading sample 8000 Loading sample 201000 Loading sample 191000 Loading sample 151000 Loading sample 71000 Loading sample 218000 Loading sample 283000 Loading sample 171000 Loading sample 47000 Loading sample 57000 Loading sample 244000 Loading sample 245000 Loading sample 211000 Loading sample 28000 Loading sample 253000 Loading sample 35000 Loading sample 205000 Loading sample 179000 Loading sample 50000 Loading sample 111000 Loading sample 85000 Loading sample 30000 Loading sample 97000 Loading sample 254000 Loading sample 10000 Loading sample 136000 Loading sample 52000 Loading sample 85000 Loading sample 1000 Loading sample 220000 Loading sample 165000 Loading sample 234000 Loading sample 162000 Loading sample 270000 Loading sample 92000 Loading sample 29000 Loading sample 105000 Loading sample 60000 Loading sample 85000 Loading sample 11000 Loading sample 8000 Loading sample 192000 Loading sample 46000 Loading sample 65000 Loading sample 166000 Loading sample 110000 Loading sample 14000 Loading sample 95000 Loading sample 149000 Loading sample 24000 Loading sample 122000 Loading sample 184000 Loading sample 266000 Loading sample 48000 Loading sample 259000 Loading sample 275000 Loading sample 65000 Loading sample 224000 Loading sample 250000 Loading sample 161000 Loading sample 128000 Loading sample 87000 Loading sample 17000 Loading sample 280000 Loading sample 152000 Loading sample 35000 Loading sample 228000 Loading sample 27000 Loading sample 209000 Loading sample 261000 Loading sample 197000 Loading sample 210000 Loading sample 260000 Loading sample 256000 Loading sample 204000 Loading sample 276000 Loading sample 266000 Loading sample 229000 Loading sample 0 Loading sample 180000 Loading sample 66000 Loading sample 198000 Loading sample 267000 Loading sample 79000 Loading sample 281000 Loading sample 234000 Loading sample 168000 Loading sample 113000 Loading sample 153000 Loading sample 59000 Loading sample 82000 Loading sample 254000 Loading sample 168000 Loading sample 201000 Loading sample 183000 Loading sample 56000 Loading sample 54000 Loading sample 116000 Loading sample 42000 Loading sample 141000 Loading sample 247000 Loading sample 201000 Loading sample 259000 Loading sample 123000 Loading sample 15000 Loading sample 235000 Loading sample 58000 Loading sample 89000 Loading sample 176000 Loading sample 117000 Loading sample 149000 Loading sample 121000 Loading sample 33000 Loading sample 118000 Loading sample 71000 Loading sample 53000 Loading sample 25000 Loading sample 180000 Loading sample 112000 Loading sample 222000 Loading sample 199000 Loading sample 37000 Loading sample 56000 Loading sample 145000 Loading sample 60000 Loading sample 187000 Loading sample 242000 Loading sample 49000 Loading sample 46000 Loading sample 251000 Loading sample 274000 Loading sample 122000 Loading sample 121000 Loading sample 19000 Loading sample 102000 Loading sample 229000 Loading sample 145000 Loading sample 35000 Loading sample 130000 Loading sample 57000 Loading sample 135000 Loading sample 169000 Loading sample 74000 Loading sample 243000 Loading sample 114000 Loading sample 255000 Loading sample 212000 Loading sample 206000 Loading sample 26000 Loading sample 212000 Loading sample 270000 Loading sample 54000 Loading sample 40000 Loading sample 95000 Loading sample 277000 Loading sample 37000 Loading sample 190000 Loading sample 175000 Loading sample 100000 Loading sample 107000 Loading sample 280000 Loading sample 13000 Loading sample 200000 Loading sample 272000 Loading sample 61000 Loading sample 92000 Loading sample 60000 Loading sample 101000 Loading sample 171000 Loading sample 23000 Loading sample 156000 Loading sample 101000 Loading sample 170000 Loading sample 258000 Loading sample 0 Loading sample 71000 Loading sample 236000 Loading sample 22000 Loading sample 7000 Loading sample 25000 Loading sample 95000 Loading sample 77000 Loading sample 85000 Loading sample 144000 Loading sample 38000 Loading sample 24000 Loading sample 87000 Loading sample 201000 Loading sample 70000 Loading sample 12000 Loading sample 100000 Loading sample 223000 Loading sample 209000 Loading sample 272000 Loading sample 233000 Loading sample 2000 Loading sample 206000 Loading sample 55000 Loading sample 110000 Loading sample 271000 Loading sample 163000 Loading sample 198000 Loading sample 109000 Loading sample 39000 Loading sample 228000 Loading sample 181000 Loading sample 231000 Loading sample 158000 Loading sample 272000 Loading sample 105000 Loading sample 92000 Loading sample 225000 Loading sample 213000 Loading sample 38000 Loading sample 258000 Loading sample 209000 Loading sample 172000 Loading sample 137000 Loading sample 187000 Loading sample 38000 Loading sample 93000 Loading sample 42000 Loading sample 53000 Loading sample 165000 Loading sample 222000 Loading sample 68000 Loading sample 224000 Loading sample 23000 Loading sample 207000 Loading sample 177000 Loading sample 108000 Loading sample 261000 Loading sample 205000 Loading sample 164000 Loading sample 132000 Loading sample 126000 Loading sample 282000 Loading sample 32000 Loading sample 263000 Loading sample 157000 Loading sample 28000 Loading sample 4000 Loading sample 103000 Loading sample 181000 Loading sample 27000 Loading sample 35000 Loading sample 100000 Loading sample 3000 Loading sample 262000 Loading sample 187000 Loading sample 148000 Loading sample 6000 Loading sample 58000 Loading sample 157000 Loading sample 120000 Loading sample 62000 Loading sample 242000 Loading sample 61000 Loading sample 145000 Loading sample 237000 Loading sample 66000 Loading sample 141000 Loading sample 54000 Loading sample 62000 Loading sample 189000 Loading sample 85000 Loading sample 39000 Loading sample 80000 Loading sample 231000 Loading sample 260000 Loading sample 121000 Loading sample 210000 Loading sample 233000 Loading sample 194000 Loading sample 204000 Loading sample 37000 Loading sample 228000 Loading sample 259000 Loading sample 129000 Loading sample 188000 Loading sample 77000 Loading sample 127000 Loading sample 278000 Loading sample 256000 Loading sample 263000 Loading sample 232000 Loading sample 242000 Loading sample 50000 Loading sample 154000 Loading sample 76000 Loading sample 199000 Loading sample 177000 Loading sample 223000 Loading sample 222000 Loading sample 0 Loading sample 209000 Loading sample 62000 Loading sample 250000 Loading sample 8000 Loading sample 161000 Loading sample 45000 Loading sample 155000 Loading sample 86000 Loading sample 261000 Loading sample 71000 Loading sample 268000 Loading sample 36000 Loading sample 209000 Loading sample 64000 Loading sample 106000 Loading sample 89000 Loading sample 8000 Loading sample 199000 Loading sample 177000 Loading sample 247000 Loading sample 134000 Loading sample 127000 Loading sample 218000 Loading sample 162000 Loading sample 84000 Loading sample 94000 Loading sample 56000 Loading sample 98000 Loading sample 196000 Loading sample 109000 Loading sample 110000 Loading sample 265000 Loading sample 52000 Loading sample 204000 Loading sample 57000 Loading sample 110000 Loading sample 225000 Loading sample 263000 Loading sample 261000 Loading sample 174000 Loading sample 239000 Loading sample 99000 Loading sample 37000 Loading sample 285000 Loading sample 199000 Loading sample 12000 Loading sample 197000 Loading sample 87000 Loading sample 251000 Loading sample 116000 Loading sample 155000 Loading sample 212000 Loading sample 84000 Loading sample 256000 Loading sample 37000 Loading sample 37000 Loading sample 45000 Loading sample 177000 Loading sample 75000 Loading sample 138000 Loading sample 210000 Loading sample 37000 Loading sample 230000 Loading sample 105000 Loading sample 213000 Loading sample 225000 Loading sample 185000 Loading sample 22000 Loading sample 10000 Loading sample 20000 Loading sample 277000 Loading sample 161000 Loading sample 213000 Loading sample 260000 Loading sample 152000 Loading sample 136000 Loading sample 126000 Loading sample 51000 Loading sample 45000 Loading sample 93000 Loading sample 154000 Loading sample 285000 Loading sample 246000 Loading sample 58000 Loading sample 211000 Loading sample 224000 Loading sample 16000 Loading sample 152000 Loading sample 266000 Loading sample 234000 Loading sample 98000 Loading sample 119000 Loading sample 243000 Loading sample 26000 Loading sample 116000 Loading sample 115000 Loading sample 185000 Loading sample 275000 Loading sample 17000 Loading sample 36000 Loading sample 141000 Loading sample 82000 Loading sample 204000 Loading sample 45000 Loading sample 73000 Loading sample 58000 Loading sample 17000 Loading sample 177000 Loading sample 201000 Loading sample 237000 Loading sample 226000 Loading sample 143000 Loading sample 11000 Loading sample 279000 Loading sample 214000 Loading sample 81000 Loading sample 106000 Loading sample 196000 Loading sample 251000 Loading sample 176000 Loading sample 189000 Loading sample 117000 Loading sample 87000 Loading sample 174000 Loading sample 197000 Loading sample 128000 Loading sample 3000 Loading sample 165000 Loading sample 263000 Loading sample 85000 Loading sample 71000 Loading sample 88000 Loading sample 83000 Loading sample 162000 Loading sample 250000 Loading sample 195000 Loading sample 189000 Loading sample 204000 Loading sample 61000 Loading sample 4000 Loading sample 103000 Loading sample 216000 Loading sample 57000 Loading sample 48000 Loading sample 248000 Loading sample 93000 Loading sample 70000 Loading sample 11000 Loading sample 56000 Loading sample 36000 Loading sample 16000 Loading sample 72000 Loading sample 155000 Loading sample 152000 Loading sample 55000 Loading sample 250000 Loading sample 230000 Loading sample 191000 Loading sample 220000 Loading sample 59000 Loading sample 102000 Loading sample 45000 Loading sample 113000 Loading sample 130000 Loading sample 67000 Loading sample 29000 Loading sample 171000 Loading sample 178000 Loading sample 103000 Loading sample 37000 Loading sample 48000 Loading sample 19000 Loading sample 257000 Loading sample 58000 Loading sample 110000 Loading sample 58000 Loading sample 42000 Loading sample 245000 Loading sample 21000 Loading sample 238000 Loading sample 27000 Loading sample 246000 Loading sample 73000 Loading sample 97000 Loading sample 267000 Loading sample 15000 Loading sample 18000 Loading sample 91000 Loading sample 103000 Loading sample 178000 Loading sample 268000 Loading sample 194000 Loading sample 46000 Loading sample 54000 Loading sample 47000 Loading sample 163000 Loading sample 202000 Loading sample 144000 Loading sample 195000 Loading sample 241000 Loading sample 56000 Loading sample 74000 Loading sample 34000 Loading sample 182000 Loading sample 57000 Loading sample 212000 Loading sample 75000 Loading sample 224000 Loading sample 94000 Loading sample 98000 Loading sample 66000 Loading sample 12000 Loading sample 10000 Loading sample 34000 Loading sample 120000 Loading sample 48000 Loading sample 169000 Loading sample 156000 Loading sample 152000 Loading sample 122000 Loading sample 243000 Loading sample 52000 Loading sample 158000 Loading sample 41000 Loading sample 31000 Loading sample 258000 Loading sample 62000 Loading sample 3000 Loading sample 197000 Loading sample 227000 Loading sample 257000 Loading sample 10000 Loading sample 257000 Loading sample 249000 Loading sample 179000 Loading sample 74000 Loading sample 174000 Loading sample 132000 Loading sample 70000 Loading sample 219000 Loading sample 173000 Loading sample 257000 Loading sample 191000 Loading sample 157000 Loading sample 117000 Loading sample 241000 Loading sample 136000 Loading sample 108000 Loading sample 169000 Loading sample 176000 Loading sample 105000 Loading sample 120000 Loading sample 136000 Loading sample 92000 Loading sample 79000 Loading sample 159000 Loading sample 121000 Loading sample 36000 Loading sample 57000 Loading sample 129000 Loading sample 86000 Loading sample 138000 Loading sample 264000 Loading sample 39000 Loading sample 96000 Loading sample 45000 Loading sample 163000 Loading sample 243000 Loading sample 185000 Loading sample 41000 Loading sample 127000 Loading sample 123000 Loading sample 68000 Loading sample 62000 Loading sample 55000 Loading sample 278000 Loading sample 268000 Loading sample 177000 Loading sample 258000 Loading sample 230000 Loading sample 89000 Loading sample 261000 Loading sample 278000 Loading sample 16000 Loading sample 110000 Loading sample 257000 Loading sample 44000 Loading sample 110000 Loading sample 177000 Loading sample 166000 Loading sample 144000 Loading sample 48000 Loading sample 140000 Loading sample 273000 Loading sample 267000 Loading sample 2000 Loading sample 54000 Loading sample 185000 Loading sample 261000 Loading sample 71000 Loading sample 113000 Loading sample 23000 Loading sample 219000 Loading sample 29000 Loading sample 201000 Loading sample 86000 Loading sample 64000 Loading sample 75000 Loading sample 261000 Loading sample 176000 Loading sample 274000 Loading sample 56000 Loading sample 47000 Loading sample 149000 Loading sample 264000 Loading sample 102000 Loading sample 79000 Loading sample 35000 Loading sample 101000 Loading sample 57000 Loading sample 138000 Loading sample 234000 Loading sample 186000 Loading sample 84000 Loading sample 86000 Loading sample 8000 Loading sample 34000 Loading sample 225000 Loading sample 208000 Loading sample 67000 Loading sample 25000 Loading sample 60000 Loading sample 35000 Loading sample 54000 Loading sample 121000 Loading sample 200000 Loading sample 241000 Loading sample 170000 Loading sample 196000 Loading sample 40000 Loading sample 220000 Loading sample 241000 Loading sample 255000 Loading sample 195000 Loading sample 10000 Loading sample 68000 Loading sample 65000 Loading sample 47000 Loading sample 115000 Loading sample 236000 Loading sample 246000 Loading sample 171000 Loading sample 158000 Loading sample 95000 Loading sample 64000 Loading sample 41000 Loading sample 76000 Loading sample 50000 Loading sample 39000 Loading sample 99000 Loading sample 100000 Loading sample 142000 Loading sample 192000 Loading sample 273000 Loading sample 48000 Loading sample 136000 Loading sample 274000 Loading sample 92000 Loading sample 259000 Loading sample 212000 Loading sample 166000 Loading sample 182000 Loading sample 195000 Loading sample 133000 Loading sample 135000 Loading sample 94000 Loading sample 85000 Loading sample 251000 Loading sample 11000 Loading sample 88000 Loading sample 188000 Loading sample 61000 Loading sample 19000 Loading sample 204000 Loading sample 267000 Loading sample 200000 Loading sample 110000 Loading sample 257000 Loading sample 75000 Loading sample 252000 Loading sample 192000 Loading sample 106000 Loading sample 146000 Loading sample 171000 Loading sample 143000 Loading sample 154000 Loading sample 54000 Loading sample 200000 Loading sample 198000 Loading sample 33000 Loading sample 87000 Loading sample 168000 Loading sample 278000 Loading sample 129000 Loading sample 77000 Loading sample 8000 Loading sample 206000 Loading sample 90000 Loading sample 144000 Loading sample 183000 Loading sample 15000 Loading sample 14000 Loading sample 166000 Loading sample 133000 Loading sample 210000 Loading sample 223000 Loading sample 257000 Loading sample 12000 Loading sample 237000 Loading sample 266000 Loading sample 233000 Loading sample 209000 Loading sample 204000 Loading sample 174000 Loading sample 37000 Loading sample 219000 Loading sample 130000 Loading sample 55000 Loading sample 115000 Loading sample 64000 Loading sample 225000 Loading sample 108000 Loading sample 284000 Loading sample 144000 Loading sample 54000 Loading sample 211000 Loading sample 228000 Loading sample 136000 Loading sample 24000 Loading sample 274000 Loading sample 277000 Loading sample 39000 Loading sample 88000 Loading sample 176000 Loading sample 209000 Loading sample 136000 Loading sample 87000 Loading sample 285000 Loading sample 119000 Loading sample 250000 Loading sample 260000 Loading sample 229000 Loading sample 156000 Loading sample 195000 Loading sample 179000 Loading sample 219000 Loading sample 44000 Loading sample 158000 Loading sample 184000 Loading sample 12000 Loading sample 2000 Loading sample 142000 Loading sample 161000 Loading sample 22000 Loading sample 41000 Loading sample 152000 Loading sample 124000 Loading sample 174000 Loading sample 26000 Loading sample 242000 Loading sample 213000 Loading sample 137000 Loading sample 260000 Loading sample 217000 Loading sample 31000 Loading sample 83000 Loading sample 103000 Loading sample 258000 Loading sample 32000 Loading sample 185000 Loading sample 53000 Loading sample 263000 Loading sample 141000 Loading sample 126000 Loading sample 166000 Loading sample 218000 Loading sample 83000 Loading sample 230000 Loading sample 235000 Loading sample 17000 Loading sample 86000 Loading sample 42000 Loading sample 105000 Loading sample 232000 Loading sample 23000 Loading sample 102000 Loading sample 183000 Loading sample 46000 Loading sample 106000 Loading sample 3000 Loading sample 134000 Loading sample 63000 Loading sample 134000 Loading sample 156000 Loading sample 76000 Loading sample 194000 Loading sample 88000 Loading sample 153000 Loading sample 149000 Loading sample 155000 Loading sample 269000 Loading sample 100000 Loading sample 33000 Loading sample 31000 Loading sample 5000 Loading sample 109000 Loading sample 273000 Loading sample 3000 Loading sample 223000 Loading sample 71000 Loading sample 231000 Loading sample 234000 Loading sample 207000 Loading sample 90000 Loading sample 42000 Loading sample 194000 Loading sample 116000 Loading sample 170000 Loading sample 122000 Loading sample 166000 Loading sample 219000 Loading sample 22000 Loading sample 227000 Loading sample 45000 Loading sample 141000 Received signal 15. Cleaning up... Performing cleanup... Performing cleanup... Warning: Error during cleanup: name '_model' is not defined Performing cleanup... Warning: Error during cleanup: name '_model' is not defined