2023-12-21 07:46:19,962 INFO StreamThr :20887 [internal.py:wandb_internal():86] W&B internal server running at pid: 20887, started at: 2023-12-21 07:46:19.959421 2023-12-21 07:46:19,966 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status 2023-12-21 07:46:21,907 INFO WriterThread:20887 [datastore.py:open_for_write():85] open: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/run-o0md2mn0.wandb 2023-12-21 07:46:21,911 DEBUG SenderThread:20887 [sender.py:send():382] send: header 2023-12-21 07:46:21,917 DEBUG SenderThread:20887 [sender.py:send():382] send: run 2023-12-21 07:46:22,539 INFO SenderThread:20887 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files 2023-12-21 07:46:22,540 INFO SenderThread:20887 [sender.py:_start_run_threads():1136] run started: o0md2mn0 with start time 1703144779.959499 2023-12-21 07:46:22,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: check_version 2023-12-21 07:46:22,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: check_version 2023-12-21 07:46:22,797 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: run_start 2023-12-21 07:46:22,811 DEBUG HandlerThread:20887 [system_info.py:__init__():32] System info init 2023-12-21 07:46:22,812 DEBUG HandlerThread:20887 [system_info.py:__init__():47] System info init done 2023-12-21 07:46:22,812 INFO HandlerThread:20887 [system_monitor.py:start():194] Starting system monitor 2023-12-21 07:46:22,812 INFO SystemMonitor:20887 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-12-21 07:46:22,812 INFO HandlerThread:20887 [system_monitor.py:probe():214] Collecting system info 2023-12-21 07:46:22,815 INFO SystemMonitor:20887 [interfaces.py:start():190] Started cpu monitoring 2023-12-21 07:46:22,816 INFO SystemMonitor:20887 [interfaces.py:start():190] Started disk monitoring 2023-12-21 07:46:22,818 INFO SystemMonitor:20887 [interfaces.py:start():190] Started gpu monitoring 2023-12-21 07:46:22,820 INFO SystemMonitor:20887 [interfaces.py:start():190] Started memory monitoring 2023-12-21 07:46:22,821 DEBUG HandlerThread:20887 [system_info.py:probe():196] Probing system 2023-12-21 07:46:22,823 INFO SystemMonitor:20887 [interfaces.py:start():190] Started network monitoring 2023-12-21 07:46:22,833 DEBUG HandlerThread:20887 [gitlib.py:_init_repo():56] git repository is invalid 2023-12-21 07:46:22,833 DEBUG HandlerThread:20887 [system_info.py:probe():244] Probing system done 2023-12-21 07:46:22,833 DEBUG HandlerThread:20887 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2023-12-21T07:46:22.821262', 'startedAt': '2023-12-21T07:46:19.914249', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'falcon_finetuning.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1Tk7cXEEpTTW4lYmAsU-XI-bUIcQ26YYq', 'host': '0dba6b8fc499', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2199.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2199.998, 'min': 0.0, 'max': 0.0}, {'current': 2199.998, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 40.18083190917969}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}} 2023-12-21 07:46:22,833 INFO HandlerThread:20887 [system_monitor.py:probe():224] Finished collecting system info 2023-12-21 07:46:22,833 INFO HandlerThread:20887 [system_monitor.py:probe():227] Publishing system info 2023-12-21 07:46:22,833 DEBUG HandlerThread:20887 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2023-12-21 07:46:22,838 DEBUG HandlerThread:20887 [system_info.py:_save_pip():68] Saving pip packages done 2023-12-21 07:46:22,842 INFO HandlerThread:20887 [system_monitor.py:probe():229] Finished publishing system info 2023-12-21 07:46:22,850 DEBUG SenderThread:20887 [sender.py:send():382] send: files 2023-12-21 07:46:22,850 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-metadata.json with policy now 2023-12-21 07:46:22,884 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:46:22,887 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:46:22,901 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:46:23,281 DEBUG SenderThread:20887 [sender.py:send():382] send: telemetry 2023-12-21 07:46:23,283 DEBUG SenderThread:20887 [sender.py:send():382] send: config 2023-12-21 07:46:23,284 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 07:46:23,284 DEBUG SenderThread:20887 [sender.py:send():382] send: telemetry 2023-12-21 07:46:23,284 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 07:46:23,285 WARNING SenderThread:20887 [sender.py:send_metric():1343] Seen metric with glob (shouldn't happen) 2023-12-21 07:46:23,285 DEBUG SenderThread:20887 [sender.py:send():382] send: telemetry 2023-12-21 07:46:23,552 INFO Thread-12 :20887 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/requirements.txt 2023-12-21 07:46:23,552 INFO Thread-12 :20887 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/output.log 2023-12-21 07:46:23,553 INFO Thread-12 :20887 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-metadata.json 2023-12-21 07:46:24,447 INFO wandb-upload_0:20887 [upload_job.py:push():131] Uploaded file /tmp/tmpvrfcfl2jwandb/qh6vmwmn-wandb-metadata.json 2023-12-21 07:46:25,311 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:46:30,312 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:46:35,313 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:46:37,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:46:37,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:46:37,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:46:41,165 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:46:46,166 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:46:51,173 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:46:51,598 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/config.yaml 2023-12-21 07:46:52,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:46:52,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:46:52,925 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:46:57,156 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:02,157 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:07,161 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:07,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:47:07,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:47:07,925 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:47:12,173 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:17,173 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:22,174 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:22,823 DEBUG SystemMonitor:20887 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2023-12-21 07:47:22,824 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:47:22,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:47:22,885 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:47:22,925 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:47:28,131 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:33,132 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:37,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:47:37,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:47:37,925 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:47:39,130 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:44,131 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:49,132 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:47:52,825 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:47:52,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:47:52,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:47:52,925 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:47:55,113 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:00,114 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:05,115 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:07,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:48:07,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:48:07,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:48:10,165 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:15,165 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:20,166 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:22,826 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:48:22,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:48:22,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:48:22,926 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:48:26,092 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:31,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:36,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:37,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:48:37,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:48:37,926 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:48:41,157 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:46,158 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:51,159 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:48:52,827 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:48:52,885 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:48:52,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:48:52,926 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:48:57,078 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:00,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 07:49:00,889 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 07:49:00,889 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 07:49:00,890 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 07:49:00,890 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 07:49:00,890 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 07:49:00,898 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 07:49:01,872 INFO Thread-12 :20887 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 07:49:02,903 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:07,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:49:07,886 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:49:07,896 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:49:08,092 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:13,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:15,966 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/output.log 2023-12-21 07:49:18,094 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:22,828 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:49:22,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:49:22,887 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:49:22,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:49:23,184 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:28,190 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:29,020 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/config.yaml 2023-12-21 07:49:33,822 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:37,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:49:37,887 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:49:37,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:49:39,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:44,094 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:49,095 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:52,829 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:49:52,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:49:52,887 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:49:52,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:49:54,144 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:49:59,145 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:04,146 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:07,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:50:07,887 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:50:07,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:50:10,112 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:15,113 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:20,113 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:22,830 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:50:22,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:50:22,887 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:50:22,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:50:26,094 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:31,095 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:36,096 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:37,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:50:37,887 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:50:37,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:50:41,096 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:46,097 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:51,098 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:50:52,831 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:50:52,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:50:52,887 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:50:52,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:50:56,122 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:01,123 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:06,123 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:07,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:51:07,888 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:51:07,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:51:11,178 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:16,178 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:21,179 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:22,832 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:51:22,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:51:22,888 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:51:22,927 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:51:27,111 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:32,111 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:37,112 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:37,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:51:37,888 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:51:37,929 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:51:42,123 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:47,123 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:52,124 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:51:52,449 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 07:51:52,450 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 07:51:52,451 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 07:51:52,457 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 07:51:52,835 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:51:53,382 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 07:51:55,217 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:51:55,219 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:51:55,220 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:51:57,458 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:02,460 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:07,461 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:10,217 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:52:10,218 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:52:10,219 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:52:12,507 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:17,528 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:22,529 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:22,835 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:52:25,217 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:52:25,219 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:52:25,220 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:52:28,481 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:33,482 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:38,483 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:40,217 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:52:40,219 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:52:40,219 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:52:44,478 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:49,480 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:52,834 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:52:54,835 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:52:55,217 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:52:55,219 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:52:55,219 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:53:00,428 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:05,429 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:10,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:53:10,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:53:10,222 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:53:10,478 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:15,479 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:20,479 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:22,835 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:53:25,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:53:25,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:53:25,222 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:53:25,499 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:30,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:35,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:40,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:53:40,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:53:40,223 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:53:41,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:46,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:51,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:53:52,836 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:53:55,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:53:55,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:53:55,223 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:53:57,422 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:02,423 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:07,424 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:10,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:54:10,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:54:10,223 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:54:12,439 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:17,442 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:17,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 07:54:17,547 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 07:54:17,548 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 07:54:17,561 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 07:54:17,732 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 07:54:22,565 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:22,836 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:54:25,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:54:25,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:54:25,223 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:54:28,481 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:33,482 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:38,483 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:40,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:54:40,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:54:40,223 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:54:43,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:48,517 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:52,842 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:54:53,846 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:54:55,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:54:55,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:54:55,223 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:54:59,465 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:04,465 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:09,466 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:10,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:55:10,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:55:10,224 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:55:14,522 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:19,523 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:22,843 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:55:24,844 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:25,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:55:25,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:55:25,224 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:55:30,440 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:35,441 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:40,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:55:40,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:55:40,224 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:55:40,472 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:45,472 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:50,473 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:55:52,843 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:55:55,221 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:55:55,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:55:55,224 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:55:55,487 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:00,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:05,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:10,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:56:10,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:56:10,224 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:56:10,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:15,502 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:20,503 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:22,844 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:56:25,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:56:25,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:56:25,224 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:56:26,473 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:31,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:36,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:40,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:56:40,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:56:40,224 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:56:42,443 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:47,444 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:52,445 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:56:52,845 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:56:55,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:56:55,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:56:55,225 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:56:57,451 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:02,452 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:07,453 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:10,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:57:10,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:57:10,225 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:57:12,163 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 07:57:12,164 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 07:57:12,164 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 07:57:12,169 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 07:57:13,157 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 07:57:13,171 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:18,172 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:22,846 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:57:23,848 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:25,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:57:25,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:57:25,224 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:57:29,435 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:34,436 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:39,437 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:40,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:57:40,230 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:57:40,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:57:44,477 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:49,478 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:52,846 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:57:54,848 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:57:55,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:57:55,230 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:57:55,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:58:00,418 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:05,418 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:10,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:58:10,230 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:58:10,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:58:10,465 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:15,466 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:20,467 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:22,847 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:58:25,222 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:58:25,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:58:25,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:58:25,482 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:30,483 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:35,484 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:40,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:58:40,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:58:40,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:58:41,442 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:46,443 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:51,443 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:58:52,848 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:58:55,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:58:55,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:58:55,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:58:56,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:01,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:06,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:10,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:59:10,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:59:10,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:59:12,458 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:17,459 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:22,460 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:22,849 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:59:25,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:59:25,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:59:25,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:59:28,437 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:33,437 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:38,438 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:40,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:59:40,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:59:40,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:59:43,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:48,477 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:52,850 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 07:59:53,850 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 07:59:55,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 07:59:55,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 07:59:55,231 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 07:59:59,414 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:04,414 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:09,415 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:10,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:00:10,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:00:10,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:00:14,462 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:19,463 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:22,850 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:00:24,852 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:25,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:00:25,231 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:00:25,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:00:30,467 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:31,444 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:00:31,446 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:00:31,450 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:00:31,467 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:00:31,633 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:00:35,473 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:40,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:00:40,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:00:40,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:00:40,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:45,514 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:50,515 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:00:52,851 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:00:55,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:00:55,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:00:55,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:00:56,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:01,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:06,514 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:10,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:01:10,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:01:10,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:01:12,477 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:17,478 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:22,479 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:22,852 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:01:25,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:01:25,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:01:25,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:01:27,483 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:32,484 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:37,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:40,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:01:40,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:01:40,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:01:43,462 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:48,462 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:52,853 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:01:53,854 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:01:55,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:01:55,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:01:55,232 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:01:59,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:04,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:09,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:10,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:02:10,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:02:10,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:02:15,422 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:20,423 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:22,854 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:02:25,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:02:25,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:02:25,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:02:25,490 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:30,491 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:35,492 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:40,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:02:40,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:02:40,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:02:40,519 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:45,520 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:50,521 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:02:52,854 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:02:55,224 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:02:55,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:02:55,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:02:56,515 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:01,516 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:06,517 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:10,225 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:03:10,232 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:03:10,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:03:12,452 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:12,620 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:03:12,623 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:03:12,624 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:03:12,628 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:03:13,020 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:03:17,630 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:22,631 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:22,858 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:03:25,225 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:03:25,233 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:03:25,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:03:28,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:33,491 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:38,495 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:40,225 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:03:40,233 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:03:40,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:03:44,447 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:49,447 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:52,858 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:03:54,860 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:03:55,225 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:03:55,233 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:03:55,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:04:00,510 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:05,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:10,225 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:04:10,233 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:04:10,233 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:04:11,492 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:16,493 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:21,493 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:22,859 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:04:25,225 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:04:25,236 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:04:25,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:04:26,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:31,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:36,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:40,225 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:04:40,236 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:04:40,236 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:04:42,429 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:47,430 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:52,431 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:04:52,860 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:04:55,226 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:04:55,236 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:04:55,236 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:04:57,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:02,502 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:07,503 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:10,226 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:05:10,236 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:05:10,236 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:05:13,422 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:18,422 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:22,861 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:05:23,862 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:25,226 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:05:25,236 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:05:25,236 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:05:29,427 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:34,428 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:39,430 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:40,226 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:05:40,236 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:05:40,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:05:44,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:49,479 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:52,862 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:05:54,864 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:05:55,226 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:05:55,236 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:05:55,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:05:55,848 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:05:55,849 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:05:55,852 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:05:55,856 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:05:56,456 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:06:00,859 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:05,860 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:10,226 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:06:10,237 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:06:10,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:06:11,438 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:16,439 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:21,440 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:22,863 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:06:25,226 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:06:25,237 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:06:25,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:06:26,497 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:31,498 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:36,505 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:40,226 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:06:40,237 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:06:40,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:06:42,480 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:47,481 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:52,482 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:06:52,865 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:06:55,227 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:06:55,237 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:06:55,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:06:58,460 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:03,461 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:08,461 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:10,227 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:07:10,237 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:07:10,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:07:14,421 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:19,422 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:22,864 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:07:24,865 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:25,227 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:07:25,237 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:07:25,237 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:07:30,500 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:35,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:40,227 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:07:40,237 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:07:40,238 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:07:40,523 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:45,524 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:50,525 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:07:52,866 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:07:55,227 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:07:55,237 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:07:55,238 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:07:56,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:01,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:06,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:10,227 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:08:10,239 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:08:10,245 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:08:12,496 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:17,496 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:22,497 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:22,866 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:08:25,227 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:08:25,239 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:08:25,240 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:08:27,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:32,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:33,738 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:08:33,739 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:08:33,740 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:08:33,747 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:08:33,854 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:08:37,753 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:42,758 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:43,253 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:08:43,255 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:08:43,255 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:08:48,496 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:52,867 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:08:53,869 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:08:58,253 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:08:58,254 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:08:58,254 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:08:59,550 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:04,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:09,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:13,253 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:09:13,254 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:09:13,254 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:09:15,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:20,502 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:22,868 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:09:25,869 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:28,253 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:09:28,254 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:09:28,254 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:09:31,464 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:36,465 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:41,466 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:43,253 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:09:43,255 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:09:43,255 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:09:46,527 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:51,528 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:52,869 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:09:56,872 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:09:58,254 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:09:58,255 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:09:58,259 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:10:02,494 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:07,495 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:12,496 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:13,254 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:10:13,255 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:10:13,255 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:10:18,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:22,870 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:10:23,871 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:28,254 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:10:28,255 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:10:28,255 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:10:29,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:34,502 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:39,502 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:43,254 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:10:43,255 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:10:43,255 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:10:45,472 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:50,473 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:52,870 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:10:55,872 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:10:58,254 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:10:58,255 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:10:58,256 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:11:01,555 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:06,555 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:10,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:11:10,388 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:11:10,389 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:11:10,393 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:11:11,271 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:11:12,402 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:15,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:11:15,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:11:15,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:11:17,773 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:22,769 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:22,871 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:11:27,876 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:30,473 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:11:30,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:11:30,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:11:33,761 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:38,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:43,763 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:45,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:11:45,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:11:45,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:11:48,785 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:52,872 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:11:53,873 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:11:58,875 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:00,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:12:00,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:12:00,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:12:04,760 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:09,761 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:14,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:15,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:12:15,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:12:15,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:12:20,750 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:22,873 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:12:25,874 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:30,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:12:30,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:12:30,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:12:31,705 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:36,706 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:41,712 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:45,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:12:45,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:12:45,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:12:47,689 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:52,690 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:12:52,874 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:12:57,875 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:00,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:13:00,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:13:00,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:13:03,779 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:08,780 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:13,781 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:15,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:13:15,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:13:15,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:13:19,686 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:22,874 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:13:24,875 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:29,876 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:30,474 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:13:30,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:13:30,486 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:13:35,700 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:40,700 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:45,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:13:45,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:13:45,487 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:13:45,757 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:47,214 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:13:47,215 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:13:47,216 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:13:47,222 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:13:47,751 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:13:51,225 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:13:52,875 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:13:56,879 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:00,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:14:00,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:14:00,487 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:14:02,745 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:07,746 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:12,747 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:15,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:14:15,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:14:15,487 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:14:17,797 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:22,797 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:22,876 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:14:27,877 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:30,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:14:30,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:14:30,487 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:14:33,764 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:38,765 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:43,765 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:45,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:14:45,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:14:45,487 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:14:49,761 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:52,879 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:14:54,880 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:14:59,881 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:00,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:15:00,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:15:00,487 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:15:05,775 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:10,776 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:15,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:15:15,487 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:15:15,488 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:15:16,751 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:21,752 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:22,878 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:15:26,879 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:30,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:15:30,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:15:30,488 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:15:32,758 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:37,759 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:42,760 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:45,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:15:45,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:15:45,488 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:15:47,776 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:52,777 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:15:52,878 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:15:57,879 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:00,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:16:00,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:16:00,488 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:16:03,681 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:08,681 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:13,682 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:15,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:16:15,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:16:15,489 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:16:18,735 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:22,879 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:16:23,880 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:28,881 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:30,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:16:30,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:16:30,489 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:16:34,182 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:16:34,184 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:16:34,184 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:16:34,184 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:34,188 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:16:34,375 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:16:39,191 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:44,192 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:45,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:16:45,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:16:45,489 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:16:49,698 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:52,880 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:16:54,881 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:16:59,882 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:00,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:17:00,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:17:00,489 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:17:05,818 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:10,819 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:15,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:17:15,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:17:15,489 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:17:16,703 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:21,704 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:22,881 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:17:26,882 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:30,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:17:30,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:17:30,489 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:17:32,738 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:37,738 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:42,739 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:45,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:17:45,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:17:45,489 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:17:48,720 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:52,881 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:17:53,882 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:17:58,883 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:00,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:18:00,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:18:00,490 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:18:04,702 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:09,703 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:14,703 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:15,476 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:18:15,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:18:15,489 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:18:19,707 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:22,882 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:18:24,883 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:29,884 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:30,477 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:18:30,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:18:30,490 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:18:35,716 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:40,717 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:45,477 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:18:45,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:18:45,490 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:18:45,744 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:50,745 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:18:52,883 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:18:55,884 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:00,477 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:19:00,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:19:00,490 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:19:01,709 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:06,709 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:11,710 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:15,477 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:19:15,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:19:15,490 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:19:17,682 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:22,682 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:22,884 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:19:23,417 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:19:23,423 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:19:23,425 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:19:23,430 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:19:23,818 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:19:28,432 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:31,103 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:19:31,145 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:19:31,145 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:19:34,414 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:39,415 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:44,415 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:46,090 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:19:46,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:19:46,104 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:19:50,307 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:19:52,884 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:19:55,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:00,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:01,090 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:20:01,103 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:20:01,104 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:20:06,369 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:11,370 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:16,091 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:20:16,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:20:16,104 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:20:17,349 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:22,349 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:22,885 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:20:27,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:31,091 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:20:31,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:20:31,104 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:20:33,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:38,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:43,392 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:46,091 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:20:46,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:20:46,104 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:20:48,411 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:52,886 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:20:53,887 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:20:58,888 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:01,091 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:21:01,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:21:01,104 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:21:04,340 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:09,341 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:14,341 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:16,091 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:21:16,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:21:16,105 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:21:19,388 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:22,892 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:21:24,897 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:29,897 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:31,091 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:21:31,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:21:31,105 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:21:35,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:40,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:45,393 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:46,091 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:21:46,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:21:46,105 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:21:51,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:21:52,888 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:21:56,889 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:01,091 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:22:01,104 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:22:01,105 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:22:02,344 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:06,438 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:22:06,439 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:22:06,440 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:22:06,452 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:22:07,378 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:22:07,455 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:12,456 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:16,094 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:22:16,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:22:16,107 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:22:18,413 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:22,888 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:22:23,890 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:28,891 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:31,092 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:22:31,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:22:31,105 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:22:34,331 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:39,332 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:44,333 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:46,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:22:46,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:22:46,105 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:22:49,376 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:52,889 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:22:54,890 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:22:59,891 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:01,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:23:01,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:23:01,106 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:23:05,351 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:10,352 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:15,352 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:16,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:23:16,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:23:16,105 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:23:20,386 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:22,890 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:23:25,891 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:30,892 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:31,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:23:31,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:23:31,106 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:23:36,407 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:41,408 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:46,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:23:46,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:23:46,106 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:23:47,399 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:52,400 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:23:52,891 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:23:57,892 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:01,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:24:01,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:24:01,106 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:24:03,316 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:08,317 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:13,318 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:16,093 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:24:16,105 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:24:16,106 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:24:18,386 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:22,891 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:24:23,892 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:28,893 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:29,997 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:24:30,000 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:24:30,002 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:24:30,006 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:24:30,793 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:24:31,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:24:31,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:24:31,532 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:24:34,844 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:39,847 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:44,847 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:46,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:24:46,532 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:24:46,533 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:24:50,753 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:24:52,892 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:24:55,893 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:00,899 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:01,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:25:01,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:25:01,539 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:25:06,739 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:11,740 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:16,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:25:16,536 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:25:16,536 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:25:16,742 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:21,743 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:22,893 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:25:26,894 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:31,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:25:31,536 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:25:31,540 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:25:32,827 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:37,827 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:42,828 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:46,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:25:46,536 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:25:46,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:25:48,820 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:52,894 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:25:53,894 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:25:58,895 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:01,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:26:01,536 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:26:01,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:26:04,739 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:09,739 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:14,740 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:16,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:26:16,536 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:26:16,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:26:19,778 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:22,894 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:26:24,895 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:29,896 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:31,530 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:26:31,536 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:26:31,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:26:35,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:40,764 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:45,765 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:46,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:26:46,536 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:26:46,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:26:51,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:26:52,895 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:26:56,896 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:01,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:27:01,536 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:27:01,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:27:02,796 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:07,796 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:12,797 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:16,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:27:16,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:27:16,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:27:18,738 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:22,268 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:27:22,269 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:27:22,270 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:27:22,276 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:27:22,899 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:27:23,182 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:27:23,900 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:28,901 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:31,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:27:31,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:27:31,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:27:34,818 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:39,820 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:44,821 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:46,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:27:46,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:27:46,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:27:50,741 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:27:52,900 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:27:55,901 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:00,902 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:01,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:28:01,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:28:01,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:28:06,805 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:11,808 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:16,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:28:16,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:28:16,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:28:16,819 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:21,820 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:22,900 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:28:26,902 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:31,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:28:31,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:28:31,537 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:28:32,752 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:37,753 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:42,755 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:46,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:28:46,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:28:46,538 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:28:47,817 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:52,818 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:28:52,901 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:28:57,902 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:01,532 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:29:01,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:29:01,538 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:29:03,791 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:08,792 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:13,793 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:16,532 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:29:16,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:29:16,538 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:29:18,793 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:22,902 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:29:23,903 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:28,904 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:31,532 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:29:31,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:29:31,538 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:29:34,761 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:39,764 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:44,765 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:46,532 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:29:46,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:29:46,538 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:29:49,817 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:52,903 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:29:54,904 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:29:59,905 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:01,532 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:30:01,538 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:30:01,538 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:30:05,831 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:07,475 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:30:07,478 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:30:07,482 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:30:07,485 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:30:07,578 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:30:11,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:16,489 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:16,550 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:30:16,568 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:30:16,569 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:30:21,802 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:22,903 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:30:26,905 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:31,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:30:31,550 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:30:31,551 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:30:32,822 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:37,823 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:42,824 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:46,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:30:46,550 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:30:46,551 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:30:48,790 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:52,904 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:30:53,911 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:30:58,912 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:01,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:31:01,550 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:31:01,551 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:31:04,787 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:09,787 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:14,788 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:16,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:31:16,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:31:16,551 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:31:20,743 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:22,905 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:31:25,909 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:30,910 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:31,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:31:31,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:31:31,554 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:31:36,856 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:41,857 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:46,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:31:46,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:31:46,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:31:47,749 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:52,750 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:31:52,906 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:31:57,907 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:01,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:32:01,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:32:01,551 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:32:03,905 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:08,905 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:13,906 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:16,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:32:16,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:32:16,551 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:32:19,825 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:22,907 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:32:24,907 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:29,908 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:31,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:32:31,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:32:31,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:32:35,784 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:40,785 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:45,786 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:46,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:32:46,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:32:46,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:32:50,822 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:32:52,907 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:32:55,908 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:00,909 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:01,542 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:33:01,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:33:01,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:33:06,807 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:09,962 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:33:09,964 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:33:09,964 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:33:09,970 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:33:10,040 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:33:11,973 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:16,543 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:33:16,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:33:16,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:33:17,841 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:22,842 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:22,908 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:33:27,909 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:31,543 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:33:31,551 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:33:31,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:33:33,826 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:38,827 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:43,828 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:46,543 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:33:46,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:33:46,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:33:49,815 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:52,909 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:33:54,910 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:33:59,911 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:01,543 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:34:01,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:34:01,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:34:05,770 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:10,771 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:15,772 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:16,543 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:34:16,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:34:16,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:34:21,756 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:22,909 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:34:26,911 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:31,544 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:34:31,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:34:31,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:34:32,829 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:37,829 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:42,830 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:46,545 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:34:46,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:34:46,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:34:47,883 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:52,884 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:34:52,910 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:34:57,911 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:01,545 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:35:01,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:35:01,552 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:35:03,841 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:08,842 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:13,842 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:16,545 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:35:16,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:35:16,553 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:35:19,783 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:22,923 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:35:24,924 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:29,925 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:31,545 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:35:31,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:35:31,553 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:35:35,818 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:40,819 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:45,820 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:46,545 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:35:46,552 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:35:46,553 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:35:50,894 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:52,912 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:35:55,913 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:35:58,074 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:35:58,075 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:35:58,075 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:35:58,082 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:35:58,446 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:36:01,086 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:01,820 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:36:01,826 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:36:01,827 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:36:06,155 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:11,156 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:16,157 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:16,721 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:36:16,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:36:16,722 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:36:22,058 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:22,912 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:36:27,914 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:31,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:36:31,727 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:36:31,763 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:36:32,995 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:37,996 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:42,997 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:46,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:36:46,723 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:36:46,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:36:48,914 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:52,913 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:36:53,916 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:36:58,917 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:01,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:37:01,723 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:37:01,763 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:37:03,949 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:08,950 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:13,951 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:16,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:37:16,723 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:37:16,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:37:19,006 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:22,914 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:37:24,915 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:29,916 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:31,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:37:31,723 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:37:31,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:37:35,916 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:40,917 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:45,917 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:46,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:37:46,723 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:37:46,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:37:50,979 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:37:52,915 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:37:56,916 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:01,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:38:01,723 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:38:01,763 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:38:01,966 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:06,967 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:11,968 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:16,722 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:38:16,723 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:38:16,763 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:38:17,965 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:22,915 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:38:23,916 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:28,917 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:31,389 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:38:31,390 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:38:31,393 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:38:31,397 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:38:31,871 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:38:32,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:38:32,384 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:38:32,387 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:38:34,592 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:39,593 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:44,594 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:47,383 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:38:47,384 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:38:47,424 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:38:50,569 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:38:52,916 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:38:55,917 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:00,918 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:02,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:39:02,384 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:39:02,425 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:39:06,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:11,601 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:16,602 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:17,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:39:17,384 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:39:17,424 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:39:21,695 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:22,917 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:39:26,918 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:31,919 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:32,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:39:32,384 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:39:32,425 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:39:37,675 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:42,675 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:47,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:39:47,385 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:39:47,425 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:39:48,572 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:52,918 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:39:53,919 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:39:58,921 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:02,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:40:02,385 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:40:02,425 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:40:04,650 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:09,650 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:14,651 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:17,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:40:17,385 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:40:17,425 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:40:20,590 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:22,919 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:40:25,920 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:30,920 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:32,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:40:32,385 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:40:32,425 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:40:36,604 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:41,604 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:46,605 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:47,385 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:40:47,385 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:40:47,426 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:40:51,643 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:40:52,919 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:40:56,920 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:01,921 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:02,384 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:41:02,385 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:41:02,426 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:41:07,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:12,623 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:14,694 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:41:14,699 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:41:14,699 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:41:14,703 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:41:15,327 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:41:17,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:41:17,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:41:17,391 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:41:17,661 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:22,662 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:22,922 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:41:27,934 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:32,389 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:41:32,389 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:41:32,390 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:41:33,621 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:38,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:43,623 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:47,389 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:41:47,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:41:47,390 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:41:49,605 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:52,927 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:41:54,928 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:41:59,929 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:02,389 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:42:02,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:42:02,390 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:42:05,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:10,628 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:15,629 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:17,389 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:42:17,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:42:17,390 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:42:20,639 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:22,928 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:42:25,929 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:30,930 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:32,389 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:42:32,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:42:32,390 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:42:36,643 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:41,644 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:46,645 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:47,393 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:42:47,393 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:42:47,394 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:42:52,641 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:42:52,928 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:42:57,930 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:02,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:43:02,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:43:02,391 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:43:03,612 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:08,613 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:13,614 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:17,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:43:17,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:43:17,398 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:43:18,746 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:22,929 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:43:23,931 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:28,931 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:32,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:43:32,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:43:32,391 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:43:34,625 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:39,626 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:44,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:47,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:43:47,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:43:47,391 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:43:49,700 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:52,930 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:43:54,931 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:43:59,932 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:02,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:44:02,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:44:02,392 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:44:05,635 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:07,573 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:44:07,574 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:44:07,575 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:44:07,578 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:44:07,833 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:44:11,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:16,583 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:17,390 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:44:17,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:44:17,391 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:44:21,690 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:22,931 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:44:26,932 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:31,933 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:32,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:44:32,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:44:32,392 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:44:37,687 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:42,688 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:47,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:44:47,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:44:47,392 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:44:47,691 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:52,692 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:44:52,931 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:44:57,933 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:02,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:45:02,392 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:45:02,392 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:45:03,638 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:08,638 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:13,639 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:17,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:45:17,392 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:45:17,393 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:45:19,635 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:22,932 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:45:24,936 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:29,938 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:32,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:45:32,392 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:45:32,392 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:45:35,668 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:40,669 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:45,669 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:47,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:45:47,392 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:45:47,392 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:45:51,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:45:52,933 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:45:56,934 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:01,936 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:02,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:46:02,392 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:46:02,393 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:46:07,654 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:12,655 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:17,391 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:46:17,392 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:46:17,392 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:46:17,709 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:22,711 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:22,941 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:46:27,943 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:31,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:46:31,513 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:46:31,513 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:46:31,519 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:46:32,353 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:46:32,670 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:46:33,173 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:46:33,173 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:46:33,419 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:38,419 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:43,429 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:47,668 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:46:47,669 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:46:47,670 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:46:49,241 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:52,941 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:46:54,942 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:46:59,943 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:02,668 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:47:02,679 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:47:02,680 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:47:05,936 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:10,937 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:15,942 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:17,668 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:47:17,670 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:47:17,671 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:47:21,888 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:22,941 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:47:26,943 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:31,944 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:32,668 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:47:32,671 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:47:32,677 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:47:37,936 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:42,937 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:47,668 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:47:47,670 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:47:47,671 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:47:48,932 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:52,942 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:47:53,943 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:47:58,944 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:02,668 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:48:02,671 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:48:02,672 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:48:04,865 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:09,870 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:14,871 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:17,668 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:48:17,670 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:48:17,670 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:48:19,886 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:22,943 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:48:24,944 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:29,945 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:32,669 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:48:32,671 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:48:32,674 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:48:35,935 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:40,936 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:45,942 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:47,669 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:48:47,671 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:48:47,672 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:48:51,877 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:48:52,944 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:48:56,945 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:01,946 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:02,669 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:49:02,671 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:49:02,672 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:49:06,949 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:11,950 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:16,950 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:17,670 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:49:17,671 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:49:17,672 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:49:22,945 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:22,945 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:49:27,946 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:31,916 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:49:31,917 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:49:31,923 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:49:31,926 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:49:31,967 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:49:33,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:49:33,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:49:33,298 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:49:33,505 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:38,506 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:43,507 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:48,295 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:49:48,296 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:49:48,336 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:49:48,592 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:52,946 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:49:53,947 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:49:58,947 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:03,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:50:03,300 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:50:03,336 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:50:04,518 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:09,519 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:14,520 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:18,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:50:18,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:50:18,336 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:50:20,488 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:22,946 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:50:25,947 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:30,948 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:33,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:50:33,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:50:33,336 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:50:36,554 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:41,554 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:46,556 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:48,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:50:48,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:50:48,336 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:50:52,507 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:50:52,947 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:50:57,948 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:02,949 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:03,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:51:03,298 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:51:03,336 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:51:08,555 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:13,556 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:18,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:51:18,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:51:18,336 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:51:18,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:22,948 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:51:23,949 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:28,949 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:33,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:51:33,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:51:33,336 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:51:34,478 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:39,479 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:44,480 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:48,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:51:48,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:51:48,337 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:51:49,593 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:52,950 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:51:54,952 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:51:59,954 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:03,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:52:03,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:52:03,337 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:52:05,573 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:08,393 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:52:08,395 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:52:08,396 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:52:08,399 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:52:08,429 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:52:11,402 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:16,403 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:18,296 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:52:18,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:52:18,300 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:52:21,583 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:22,951 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:52:26,959 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:31,960 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:33,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:52:33,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:52:33,338 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:52:37,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:42,514 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:47,514 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:48,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:52:48,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:52:48,337 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:52:52,570 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:52:52,952 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:52:57,953 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:02,954 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:03,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:53:03,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:53:03,337 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:53:08,531 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:13,532 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:18,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:53:18,297 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:53:18,337 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:53:18,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:22,952 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:53:23,955 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:28,956 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:33,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:53:33,298 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:53:33,337 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:53:34,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:39,501 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:44,502 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:48,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:53:48,298 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:53:48,338 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:53:49,571 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:52,953 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:53:54,954 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:53:59,955 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:03,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:54:03,298 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:54:03,338 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:54:05,534 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:10,535 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:15,537 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:18,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:54:18,298 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:54:18,338 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:54:20,625 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:22,954 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:54:25,955 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:30,956 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:33,297 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:54:33,298 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:54:33,338 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:54:36,590 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:41,591 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:46,592 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:48,298 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:54:48,298 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:54:48,338 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:54:52,487 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:54:52,955 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:54:56,187 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:54:56,191 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:54:56,192 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:54:56,195 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:54:56,953 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:54:58,197 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:03,202 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:04,596 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:55:04,597 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:55:04,598 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:55:08,856 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:13,857 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:18,858 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:19,507 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:55:19,508 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:55:19,597 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:55:22,955 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:55:23,956 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:28,957 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:33,958 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:34,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:55:34,508 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:55:34,597 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:55:39,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:44,763 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:49,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:55:49,508 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:55:49,597 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:55:50,724 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:55:52,956 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:55:55,957 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:00,958 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:04,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:56:04,508 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:56:04,598 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:56:06,792 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:11,793 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:16,794 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:19,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:56:19,508 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:56:19,598 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:56:22,712 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:22,957 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:56:27,959 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:32,960 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:34,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:56:34,509 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:56:34,598 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:56:38,709 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:43,710 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:48,710 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:49,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:56:49,509 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:56:49,598 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:56:52,958 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:56:53,959 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:56:58,960 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:03,960 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:04,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:57:04,509 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:57:04,598 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:57:09,795 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:14,796 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:19,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:57:19,509 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:57:19,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:57:20,751 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:22,967 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:57:25,968 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:30,969 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:34,508 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:57:34,509 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:57:34,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:57:36,211 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 08:57:36,215 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 08:57:36,215 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 08:57:36,215 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:36,219 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 08:57:36,353 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 08:57:41,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:46,223 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:49,510 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:57:49,511 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:57:49,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:57:51,789 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:57:52,968 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:57:56,970 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:01,970 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:04,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:58:04,511 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:58:04,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:58:07,815 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:12,816 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:17,816 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:19,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:58:19,511 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:58:19,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:58:22,839 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:22,969 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:58:27,970 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:32,971 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:34,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:58:34,511 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:58:34,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:58:38,757 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:43,758 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:48,758 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:49,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:58:49,511 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:58:49,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:58:52,969 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:58:53,970 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:58:58,971 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:03,972 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:04,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:59:04,511 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:59:04,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:59:09,800 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:14,801 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:19,511 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:59:19,512 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:59:19,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:59:20,775 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:22,970 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:59:25,971 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:30,972 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:34,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:59:34,513 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:59:34,599 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:59:36,723 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:41,724 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:46,725 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:49,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 08:59:49,512 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 08:59:49,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 08:59:51,765 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 08:59:52,972 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 08:59:56,973 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:01,974 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:04,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:00:04,512 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:00:04,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:00:07,778 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:12,779 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:17,780 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:19,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:00:19,512 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:00:19,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:00:22,972 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:00:22,973 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:27,973 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:32,974 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:34,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:00:34,512 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:00:34,607 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:00:38,719 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:43,720 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:48,720 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:49,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:00:49,512 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:00:49,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:00:51,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 09:00:51,517 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 09:00:51,517 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 09:00:51,522 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 09:00:51,921 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 09:00:52,972 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:00:53,976 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:00:58,977 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:03,987 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:04,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:01:04,513 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:01:04,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:01:09,747 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:14,748 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:19,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:01:19,513 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:01:19,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:01:19,773 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:22,976 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:01:24,979 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:29,979 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:34,512 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:01:34,513 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:01:34,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:01:35,712 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:40,713 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:45,713 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:49,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:01:49,513 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:01:49,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:01:50,742 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:01:52,974 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:01:55,975 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:00,976 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:04,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:02:04,513 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:02:04,600 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:02:06,756 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:11,756 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:16,757 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:19,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:02:19,513 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:02:19,601 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:02:21,768 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:22,975 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:02:26,978 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:31,978 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:34,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:02:34,515 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:02:34,601 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:02:37,778 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:42,779 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:47,780 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:49,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:02:49,514 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:02:49,601 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:02:52,797 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:02:52,975 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:02:57,979 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:02,981 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:04,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:03:04,515 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:03:04,601 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:03:08,758 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:13,758 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:18,759 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:19,513 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:03:19,514 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:03:19,601 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:03:22,976 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:03:23,977 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:27,420 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 09:03:27,422 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 09:03:27,424 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 09:03:27,430 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 09:03:27,440 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 09:03:29,432 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:34,432 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:34,571 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:03:34,573 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:03:34,606 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:03:39,857 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:44,858 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:49,571 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:03:49,571 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:03:49,611 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:03:49,860 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:52,977 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:03:54,978 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:03:59,979 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:04,571 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:04:04,571 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:04:04,612 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:04:05,849 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:10,849 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:15,850 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:19,571 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:04:19,571 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:04:19,611 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:04:21,843 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:22,977 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:04:26,979 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:31,980 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:34,571 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:04:34,572 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:04:34,611 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:04:37,786 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:42,787 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:47,788 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:49,571 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:04:49,572 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:04:49,611 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:04:52,858 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:04:52,978 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:04:57,979 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:02,980 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:04,571 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:05:04,572 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:05:04,612 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:05:08,856 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:13,856 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:18,857 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:19,573 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:05:19,575 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:05:19,614 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:05:22,979 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:05:23,980 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:28,981 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:33,990 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:34,573 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:05:34,574 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:05:34,614 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:05:39,784 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:44,785 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:49,573 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:05:49,574 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:05:49,614 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:05:49,854 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:52,980 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:05:54,981 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:05:59,982 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:02,486 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 09:06:02,488 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 09:06:02,488 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 09:06:02,492 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 09:06:02,831 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 09:06:04,578 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:06:04,580 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:06:04,628 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:06:05,795 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:10,796 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:15,797 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:19,579 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:06:19,580 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:06:19,624 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:06:21,778 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:22,981 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:06:26,982 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:31,984 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:34,579 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:06:34,579 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:06:34,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:06:37,775 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:42,776 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:47,777 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:49,579 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:06:49,579 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:06:49,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:06:52,790 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:06:52,981 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:06:57,982 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:02,984 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:04,579 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:07:04,579 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:07:04,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:07:08,836 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:13,837 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:18,837 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:19,579 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:07:19,580 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:07:19,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:07:22,982 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:07:23,983 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:28,984 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:33,985 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:34,579 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:07:34,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:07:34,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:07:39,873 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:44,873 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:49,579 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:07:49,580 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:07:49,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:07:50,859 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:07:52,983 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:07:55,984 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:00,985 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:04,580 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:08:04,580 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:08:04,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:08:06,827 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:11,827 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:16,837 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:19,580 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:08:19,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:08:19,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:08:21,880 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:22,984 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:08:26,985 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:31,986 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:34,580 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:08:34,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:08:34,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:08:37,850 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:42,851 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:47,851 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:49,580 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:08:49,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:08:49,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:08:52,456 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 09:08:52,459 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 09:08:52,459 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 09:08:52,465 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 09:08:52,985 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:08:52,989 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:08:53,341 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 09:08:57,988 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:02,994 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:04,580 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:09:04,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:09:04,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:09:08,832 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:13,833 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:18,833 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:19,580 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:09:19,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:09:19,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:09:22,986 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:09:23,987 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:28,987 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:33,988 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:34,580 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:09:34,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:09:34,626 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:09:39,814 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:44,815 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:49,580 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:09:49,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:09:49,626 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:09:50,787 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:09:52,988 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:09:55,989 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:00,990 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:04,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:10:04,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:10:04,626 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:10:06,864 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:11,864 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:16,865 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:19,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:10:19,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:10:19,626 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:10:22,799 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:22,987 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:10:27,989 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:32,989 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:34,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:10:34,581 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:10:34,626 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:10:38,802 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:43,803 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:48,804 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:49,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:10:49,582 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:10:49,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:10:52,988 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:10:53,989 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:10:58,990 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:03,991 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:04,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:11:04,582 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:11:04,634 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:11:09,880 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:14,880 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:19,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:11:19,582 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:11:19,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:11:20,812 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:22,989 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:11:25,990 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:30,991 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:34,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:11:34,582 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:11:34,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:11:36,482 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 09:11:36,485 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 09:11:36,485 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 09:11:36,485 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:36,501 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 09:11:36,751 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 09:11:41,506 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:46,506 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:49,581 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:11:49,582 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:11:49,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:11:51,921 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:11:52,990 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:11:56,991 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:01,992 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:04,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:12:04,582 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:12:04,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:12:07,779 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:12,780 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:17,780 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:19,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:12:19,582 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:12:19,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:12:22,819 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:22,990 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:12:27,992 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:32,992 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:34,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:12:34,582 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:12:34,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:12:38,805 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:43,806 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:48,807 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:49,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:12:49,583 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:12:49,627 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:12:52,991 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:12:53,992 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:12:58,993 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:03,994 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:04,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:13:04,583 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:13:04,628 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:13:09,762 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:14,763 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:19,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:13:19,583 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:13:19,628 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:13:19,824 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:22,992 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:13:24,993 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:29,994 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:34,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:13:34,583 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:13:34,628 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:13:35,823 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:40,824 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:45,825 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:49,582 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:13:49,583 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:13:49,628 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:13:50,848 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:13:52,993 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:13:55,994 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:00,995 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:04,583 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:14:04,583 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:14:04,628 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:14:06,847 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:10,388 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 09:14:10,390 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 09:14:10,391 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 09:14:10,396 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 09:14:11,170 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 09:14:12,400 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:17,401 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:21,394 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:14:21,395 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:14:21,398 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:14:21,660 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: partial_history 2023-12-21 09:14:21,673 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 09:14:21,679 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 09:14:21,680 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 09:14:21,681 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 09:14:21,681 DEBUG SenderThread:20887 [sender.py:send():382] send: metric 2023-12-21 09:14:21,682 DEBUG SenderThread:20887 [sender.py:send():382] send: history 2023-12-21 09:14:21,682 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: summary_record 2023-12-21 09:14:21,691 INFO SenderThread:20887 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2023-12-21 09:14:22,214 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/wandb-summary.json 2023-12-21 09:14:22,692 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:22,993 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:14:27,999 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:29,234 INFO Thread-12 :20887 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/falcon-7b-sharded-bf16-finetuned-mental-health-conversational/wandb/run-20231221_074619-o0md2mn0/files/config.yaml 2023-12-21 09:14:33,324 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:36,394 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:14:36,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:14:36,396 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:14:38,616 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:43,617 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:48,618 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:51,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:14:51,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:14:51,396 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:14:52,994 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:14:53,995 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:14:55,172 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: pause 2023-12-21 09:14:55,174 INFO HandlerThread:20887 [handler.py:handle_request_pause():705] stopping system metrics thread 2023-12-21 09:14:55,174 INFO HandlerThread:20887 [system_monitor.py:finish():203] Stopping system monitor 2023-12-21 09:14:55,174 DEBUG SystemMonitor:20887 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2023-12-21 09:14:55,175 DEBUG SystemMonitor:20887 [system_monitor.py:_start():183] Publishing last batch of metrics 2023-12-21 09:14:55,174 INFO HandlerThread:20887 [interfaces.py:finish():202] Joined cpu monitor 2023-12-21 09:14:55,176 INFO HandlerThread:20887 [interfaces.py:finish():202] Joined disk monitor 2023-12-21 09:14:55,195 INFO HandlerThread:20887 [interfaces.py:finish():202] Joined gpu monitor 2023-12-21 09:14:55,196 INFO HandlerThread:20887 [interfaces.py:finish():202] Joined memory monitor 2023-12-21 09:14:55,197 INFO HandlerThread:20887 [interfaces.py:finish():202] Joined network monitor 2023-12-21 09:14:55,197 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:14:55,648 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: resume 2023-12-21 09:14:55,649 INFO HandlerThread:20887 [handler.py:handle_request_resume():696] starting system metrics thread 2023-12-21 09:14:55,649 INFO HandlerThread:20887 [system_monitor.py:start():194] Starting system monitor 2023-12-21 09:14:55,649 INFO SystemMonitor:20887 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-12-21 09:14:55,652 INFO SystemMonitor:20887 [interfaces.py:start():190] Started cpu monitoring 2023-12-21 09:14:55,655 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: pause 2023-12-21 09:14:55,656 INFO HandlerThread:20887 [handler.py:handle_request_pause():705] stopping system metrics thread 2023-12-21 09:14:55,656 INFO HandlerThread:20887 [system_monitor.py:finish():203] Stopping system monitor 2023-12-21 09:14:55,656 INFO HandlerThread:20887 [interfaces.py:finish():202] Joined cpu monitor 2023-12-21 09:14:55,656 WARNING HandlerThread:20887 [interfaces.py:finish():207] Failed to finish disk monitoring: cannot join thread before it is started 2023-12-21 09:14:55,663 INFO SystemMonitor:20887 [interfaces.py:start():190] Started disk monitoring 2023-12-21 09:14:55,663 DEBUG SystemMonitor:20887 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2023-12-21 09:14:55,663 DEBUG SystemMonitor:20887 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2023-12-21 09:14:55,663 DEBUG SystemMonitor:20887 [system_monitor.py:_start():183] Publishing last batch of metrics 2023-12-21 09:14:55,665 DEBUG SenderThread:20887 [sender.py:send():382] send: stats 2023-12-21 09:14:59,666 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:04,667 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:06,394 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:15:06,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:15:06,395 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:15:10,593 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:15,594 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:20,595 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:21,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:15:21,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:15:21,395 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:15:26,588 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:31,589 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:36,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:15:36,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:15:36,396 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:15:36,686 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:41,687 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:46,688 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:51,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:15:51,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:15:51,396 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:15:52,682 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:15:57,683 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:02,684 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:06,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:16:06,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:16:06,396 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:16:08,637 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:13,637 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:18,638 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:21,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:16:21,397 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:16:21,397 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:16:23,728 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:28,729 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:33,730 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:36,395 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:16:36,396 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:16:36,437 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:16:39,642 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:44,643 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:49,644 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:16:51,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:16:51,396 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:16:51,436 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:16:55,621 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:00,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:05,623 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:06,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:17:06,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:17:06,397 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:17:10,640 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:15,641 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:20,642 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:21,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:17:21,397 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:17:21,437 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:17:26,633 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:31,633 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:36,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:17:36,397 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:17:36,398 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:17:36,663 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:41,664 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:46,667 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:51,396 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:17:51,397 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:17:51,398 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:17:52,621 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:17:57,622 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:18:02,624 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report 2023-12-21 09:18:04,209 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: resume 2023-12-21 09:18:04,212 INFO HandlerThread:20887 [handler.py:handle_request_resume():696] starting system metrics thread 2023-12-21 09:18:04,212 INFO HandlerThread:20887 [system_monitor.py:start():194] Starting system monitor 2023-12-21 09:18:04,213 INFO SystemMonitor:20887 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-12-21 09:18:04,226 INFO SystemMonitor:20887 [interfaces.py:start():190] Started cpu monitoring 2023-12-21 09:18:04,245 INFO SystemMonitor:20887 [interfaces.py:start():190] Started disk monitoring 2023-12-21 09:18:04,265 INFO SystemMonitor:20887 [interfaces.py:start():190] Started gpu monitoring 2023-12-21 09:18:04,271 INFO SystemMonitor:20887 [interfaces.py:start():190] Started memory monitoring 2023-12-21 09:18:04,298 INFO SystemMonitor:20887 [interfaces.py:start():190] Started network monitoring 2023-12-21 09:18:06,432 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: internal_messages 2023-12-21 09:18:06,435 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: stop_status 2023-12-21 09:18:06,436 DEBUG SenderThread:20887 [sender.py:send_request():409] send_request: stop_status 2023-12-21 09:18:07,709 DEBUG HandlerThread:20887 [handler.py:handle_request():146] handle_request: status_report