| [2023-05-13 16:54:01] INFO - super_gradients.common.crash_handler.crash_tips_setup - Crash tips is enabled. You can set your environment variable to CRASH_HANDLER=FALSE to disable it | |
| [2023-05-13 16:54:06] DEBUG - pip._internal.vcs.versioncontrol - Registered VCS backend: bzr | |
| [2023-05-13 16:54:06] DEBUG - pip._internal.vcs.versioncontrol - Registered VCS backend: git | |
| [2023-05-13 16:54:06] DEBUG - pip._internal.vcs.versioncontrol - Registered VCS backend: hg | |
| [2023-05-13 16:54:06] DEBUG - pip._internal.vcs.versioncontrol - Registered VCS backend: svn | |
| [2023-05-13 16:54:06] DEBUG - super_gradients.common.sg_loggers.clearml_sg_logger - Failed to import clearml | |
| [2023-05-13 16:54:06] DEBUG - super_gradients.modules - Failed to import pytorch_quantization: cannot import name 'Bottleneck' from partially initialized module 'super_gradients.training.models.classification_models.resnet' (most likely due to a circular import) (/usr/local/lib/python3.10/dist-packages/super_gradients/training/models/classification_models/resnet.py) | |
| [2023-05-13 16:54:06] DEBUG - hydra.core.utils - Setting JobRuntime:name=UNKNOWN_NAME | |
| [2023-05-13 16:54:06] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 16:54:06] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 16:54:07] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 16:54:07] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 16:54:07] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 16:54:07] WARNING - super_gradients.training.utils.quantization - Failed to import pytorch_quantization | |
| [2023-05-13 16:54:07] WARNING - super_gradients.training.utils.quantization.calibrator - Failed to import pytorch_quantization | |
| [2023-05-13 16:54:07] WARNING - super_gradients.training.utils.quantization.export - Failed to import pytorch_quantization | |
| [2023-05-13 16:54:07] WARNING - super_gradients.training.utils.quantization.selective_quantization_utils - Failed to import pytorch_quantization | |
| [2023-05-13 16:54:07] DEBUG - super_gradients.training.qat_trainer.qat_trainer - Failed to import pytorch_quantization: | |
| [2023-05-13 16:54:07] DEBUG - super_gradients.training.qat_trainer.qat_trainer - name 'QuantizedMetadata' is not defined | |
| [2023-05-13 16:54:07] DEBUG - super_gradients.sanity_check.env_sanity_check - pyparsing==2.4.7 does not satisfy requirement pyparsing==2.4.5 | |
| [2023-05-13 16:54:33] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): app.roboflow.com:443 | |
| [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - https://app.roboflow.com:443 "GET /query/cliAuthToken/36509e31-f8df-4c6c-9182-357addecb9cd HTTP/1.1" 200 145 | |
| [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 | |
| [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "POST /?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 182 | |
| [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 | |
| [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "POST /?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 182 | |
| [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 | |
| [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 3338 | |
| [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 | |
| [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon/new-pothole-detection?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 7291 | |
| [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 | |
| [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon/new-pothole-detection?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 7291 | |
| [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 | |
| [2023-05-13 16:54:36] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon/new-pothole-detection/2?nocache=true&api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 4611 | |
| [2023-05-13 16:54:36] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 | |
| [2023-05-13 16:54:36] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon/new-pothole-detection/2/yolov5pytorch?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 3494 | |
| [2023-05-13 17:01:06] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 17:01:08] WARNING - super_gradients.training.datasets.detection_datasets.detection_dataset - Found 1090 invalid bbox that were ignored. For more information, please set `show_all_warnings=True`. | |
| [2023-05-13 17:01:08] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 17:01:08] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 17:01:09] WARNING - super_gradients.training.datasets.detection_datasets.detection_dataset - Found 4 invalid bbox that were ignored. For more information, please set `show_all_warnings=True`. | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: Matching sans\-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0. | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-Italic.ttf', name='DejaVu Serif', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUni.ttf', name='STIXNonUnicode', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-Oblique.ttf', name='DejaVu Sans Mono', style='oblique', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizOneSymReg.ttf', name='STIXSizeOneSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansDisplay.ttf', name='DejaVu Sans Display', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFiveSymReg.ttf', name='STIXSizeFiveSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizOneSymBol.ttf', name='STIXSizeOneSym', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmex10.ttf', name='cmex10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerifDisplay.ttf', name='DejaVu Serif Display', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFourSymReg.ttf', name='STIXSizeFourSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniIta.ttf', name='STIXNonUnicode', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizThreeSymBol.ttf', name='STIXSizeThreeSym', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmmi10.ttf', name='cmmi10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmsy10.ttf', name='cmsy10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniBolIta.ttf', name='STIXNonUnicode', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizTwoSymBol.ttf', name='STIXSizeTwoSym', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniBol.ttf', name='STIXNonUnicode', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralBolIta.ttf', name='STIXGeneral', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmb10.ttf', name='cmb10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizThreeSymReg.ttf', name='STIXSizeThreeSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmtt10.ttf', name='cmtt10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmss10.ttf', name='cmss10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-Bold.ttf', name='DejaVu Sans Mono', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFourSymBol.ttf', name='STIXSizeFourSym', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralBol.ttf', name='STIXGeneral', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono.ttf', name='DejaVu Sans Mono', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-Oblique.ttf', name='DejaVu Sans', style='oblique', variant='normal', weight=400, stretch='normal', size='scalable')) = 1.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-BoldOblique.ttf', name='DejaVu Sans Mono', style='oblique', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralItalic.ttf', name='STIXGeneral', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmr10.ttf', name='cmr10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif.ttf', name='DejaVu Serif', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf', name='DejaVu Sans', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 0.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizTwoSymReg.ttf', name='STIXSizeTwoSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneral.ttf', name='STIXGeneral', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-BoldItalic.ttf', name='DejaVu Serif', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-Bold.ttf', name='DejaVu Serif', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-BoldOblique.ttf', name='DejaVu Sans', style='oblique', variant='normal', weight=700, stretch='normal', size='scalable')) = 1.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-Bold.ttf', name='DejaVu Sans', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 0.33499999999999996 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Regular.ttf', name='Liberation Sans Narrow', style='normal', variant='normal', weight=400, stretch='condensed', size='scalable')) = 10.25 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationMono-Italic.ttf', name='Liberation Mono', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSansNarrow-BoldItalic.ttf', name='Liberation Sans Narrow', style='italic', variant='normal', weight=700, stretch='condensed', size='scalable')) = 11.535 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Bold.ttf', name='Liberation Sans Narrow', style='normal', variant='normal', weight=700, stretch='condensed', size='scalable')) = 10.535 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationMono-Bold.ttf', name='Liberation Mono', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSans-Italic.ttf', name='Liberation Sans', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSerif-Italic.ttf', name='Liberation Serif', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationMono-BoldItalic.ttf', name='Liberation Mono', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf', name='Liberation Sans', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Italic.ttf', name='Liberation Sans Narrow', style='italic', variant='normal', weight=400, stretch='condensed', size='scalable')) = 11.25 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSerif-BoldItalic.ttf', name='Liberation Serif', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSerif-Bold.ttf', name='Liberation Serif', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSans-BoldItalic.ttf', name='Liberation Sans', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationMono-Regular.ttf', name='Liberation Mono', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSerif-Regular.ttf', name='Liberation Serif', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/humor-sans/Humor-Sans.ttf', name='Humor Sans', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSans-Bold.ttf', name='Liberation Sans', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 | |
| [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: Matching sans\-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0 to DejaVu Sans ('/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000. | |
| [2023-05-13 17:03:47] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 17:03:48] WARNING - super_gradients.training.datasets.detection_datasets.detection_dataset - Found 1090 invalid bbox that were ignored. For more information, please set `show_all_warnings=True`. | |
| [2023-05-13 17:03:48] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 17:03:49] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 17:03:49] WARNING - super_gradients.training.datasets.detection_datasets.detection_dataset - Found 4 invalid bbox that were ignored. For more information, please set `show_all_warnings=True`. | |
| [2023-05-13 17:04:34] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 17:04:36] INFO - super_gradients.training.utils.checkpoint_utils - License Notification: YOLO-NAS pre-trained weights are subjected to the specific license terms and conditions detailed in | |
| https://github.com/Deci-AI/super-gradients/blob/master/LICENSE.YOLONAS.md | |
| By downloading the pre-trained weight files you agree to comply with these terms. | |
| [2023-05-13 17:05:36] INFO - super_gradients.training.sg_trainer.sg_trainer - Using EMA with params {'decay': 0.9, 'decay_type': 'threshold'} | |
| [2023-05-13 17:05:40] DEBUG - tensorflow - Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client. | |
| [2023-05-13 17:05:40] DEBUG - h5py._conv - Creating converter from 7 to 5 | |
| [2023-05-13 17:05:40] DEBUG - h5py._conv - Creating converter from 5 to 7 | |
| [2023-05-13 17:05:40] DEBUG - h5py._conv - Creating converter from 7 to 5 | |
| [2023-05-13 17:05:40] DEBUG - h5py._conv - Creating converter from 5 to 7 | |
| [2023-05-13 17:05:41] DEBUG - jaxlib.mlir._mlir_libs - Initializing MLIR with module: _site_initialize_0 | |
| [2023-05-13 17:05:41] DEBUG - jaxlib.mlir._mlir_libs - Registering dialects from initializer <module 'jaxlib.mlir._mlir_libs._site_initialize_0' from '/usr/local/lib/python3.10/dist-packages/jaxlib/mlir/_mlir_libs/_site_initialize_0.so'> | |
| [2023-05-13 17:05:41] DEBUG - jax._src.path - etils.epath found. Using etils.epath for file I/O. | |
| [2023-05-13 17:05:42] INFO - numexpr.utils - NumExpr defaulting to 2 threads. | |
| [2023-05-13 17:05:49] INFO - super_gradients.training.utils.sg_trainer_utils - TRAINING PARAMETERS: | |
| - Mode: Single GPU | |
| - Number of GPUs: 1 (1 available on the machine) | |
| - Dataset size: 6086 (len(train_set)) | |
| - Batch size per GPU: 16 (batch_size) | |
| - Batch Accumulate: 1 (batch_accumulate) | |
| - Total batch size: 16 (num_gpus * batch_size) | |
| - Effective Batch size: 16 (num_gpus * batch_size * batch_accumulate) | |
| - Iterations per epoch: 380 (len(train_loader)) | |
| - Gradient updates per epoch: 380 (len(train_loader) / batch_accumulate) | |
| [2023-05-13 17:13:26] INFO - super_gradients.training.sg_trainer.sg_trainer - | |
| [MODEL TRAINING EXECUTION HAS BEEN INTERRUPTED]... Please wait until SOFT-TERMINATION process finishes and saves all of the Model Checkpoints and log files before terminating... | |
| [2023-05-13 17:13:26] INFO - super_gradients.training.sg_trainer.sg_trainer - For HARD Termination - Stop the process again | |
| [2023-05-13 17:13:26] INFO - super_gradients.common.sg_loggers.base_sg_logger - [CLEANUP] - Successfully stopped system monitoring process | |
| [2023-05-13 17:13:27] DEBUG - hydra.core.utils - Setting JobRuntime:name=app | |
| [2023-05-13 17:13:57] INFO - super_gradients.training.sg_trainer.sg_trainer - Using EMA with params {'decay': 0.9, 'decay_type': 'threshold'} | |
| [2023-05-13 17:13:58] DEBUG - super_gradients.training.utils.sg_trainer_utils - "events.out.tfevents.1683998007.cf6772931124.3382.1" will not be deleted | |
| [2023-05-13 17:13:58] DEBUG - super_gradients.training.utils.sg_trainer_utils - "events.out.tfevents.1683997543.cf6772931124.3382.0" will not be deleted | |