File size: 25,393 Bytes
5537b6c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
[2023-05-13 16:54:01] INFO - super_gradients.common.crash_handler.crash_tips_setup - Crash tips is enabled. You can set your environment variable to CRASH_HANDLER=FALSE to disable it [2023-05-13 16:54:06] DEBUG - pip._internal.vcs.versioncontrol - Registered VCS backend: bzr [2023-05-13 16:54:06] DEBUG - pip._internal.vcs.versioncontrol - Registered VCS backend: git [2023-05-13 16:54:06] DEBUG - pip._internal.vcs.versioncontrol - Registered VCS backend: hg [2023-05-13 16:54:06] DEBUG - pip._internal.vcs.versioncontrol - Registered VCS backend: svn [2023-05-13 16:54:06] DEBUG - super_gradients.common.sg_loggers.clearml_sg_logger - Failed to import clearml [2023-05-13 16:54:06] DEBUG - super_gradients.modules - Failed to import pytorch_quantization: cannot import name 'Bottleneck' from partially initialized module 'super_gradients.training.models.classification_models.resnet' (most likely due to a circular import) (/usr/local/lib/python3.10/dist-packages/super_gradients/training/models/classification_models/resnet.py) [2023-05-13 16:54:06] DEBUG - hydra.core.utils - Setting JobRuntime:name=UNKNOWN_NAME [2023-05-13 16:54:06] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 16:54:06] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 16:54:07] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 16:54:07] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 16:54:07] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 16:54:07] WARNING - super_gradients.training.utils.quantization - Failed to import pytorch_quantization [2023-05-13 16:54:07] WARNING - super_gradients.training.utils.quantization.calibrator - Failed to import pytorch_quantization [2023-05-13 16:54:07] WARNING - super_gradients.training.utils.quantization.export - Failed to import pytorch_quantization [2023-05-13 16:54:07] WARNING - super_gradients.training.utils.quantization.selective_quantization_utils - Failed to import pytorch_quantization [2023-05-13 16:54:07] DEBUG - super_gradients.training.qat_trainer.qat_trainer - Failed to import pytorch_quantization: [2023-05-13 16:54:07] DEBUG - super_gradients.training.qat_trainer.qat_trainer - name 'QuantizedMetadata' is not defined [2023-05-13 16:54:07] DEBUG - super_gradients.sanity_check.env_sanity_check - pyparsing==2.4.7 does not satisfy requirement pyparsing==2.4.5 [2023-05-13 16:54:33] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): app.roboflow.com:443 [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - https://app.roboflow.com:443 "GET /query/cliAuthToken/36509e31-f8df-4c6c-9182-357addecb9cd HTTP/1.1" 200 145 [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "POST /?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 182 [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "POST /?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 182 [2023-05-13 16:54:34] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 3338 [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon/new-pothole-detection?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 7291 [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon/new-pothole-detection?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 7291 [2023-05-13 16:54:35] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 [2023-05-13 16:54:36] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon/new-pothole-detection/2?nocache=true&api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 4611 [2023-05-13 16:54:36] DEBUG - urllib3.connectionpool - Starting new HTTPS connection (1): api.roboflow.com:443 [2023-05-13 16:54:36] DEBUG - urllib3.connectionpool - https://api.roboflow.com:443 "GET /smartathon/new-pothole-detection/2/yolov5pytorch?api_key=2NdQm1ivtFCAYiOLVTwn HTTP/1.1" 200 3494 [2023-05-13 17:01:06] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 17:01:08] WARNING - super_gradients.training.datasets.detection_datasets.detection_dataset - Found 1090 invalid bbox that were ignored. For more information, please set `show_all_warnings=True`. [2023-05-13 17:01:08] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 17:01:08] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 17:01:09] WARNING - super_gradients.training.datasets.detection_datasets.detection_dataset - Found 4 invalid bbox that were ignored. For more information, please set `show_all_warnings=True`. [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: Matching sans\-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0. [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-Italic.ttf', name='DejaVu Serif', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUni.ttf', name='STIXNonUnicode', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-Oblique.ttf', name='DejaVu Sans Mono', style='oblique', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizOneSymReg.ttf', name='STIXSizeOneSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansDisplay.ttf', name='DejaVu Sans Display', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFiveSymReg.ttf', name='STIXSizeFiveSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizOneSymBol.ttf', name='STIXSizeOneSym', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmex10.ttf', name='cmex10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerifDisplay.ttf', name='DejaVu Serif Display', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFourSymReg.ttf', name='STIXSizeFourSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniIta.ttf', name='STIXNonUnicode', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizThreeSymBol.ttf', name='STIXSizeThreeSym', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmmi10.ttf', name='cmmi10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmsy10.ttf', name='cmsy10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniBolIta.ttf', name='STIXNonUnicode', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizTwoSymBol.ttf', name='STIXSizeTwoSym', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniBol.ttf', name='STIXNonUnicode', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralBolIta.ttf', name='STIXGeneral', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmb10.ttf', name='cmb10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizThreeSymReg.ttf', name='STIXSizeThreeSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmtt10.ttf', name='cmtt10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmss10.ttf', name='cmss10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-Bold.ttf', name='DejaVu Sans Mono', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFourSymBol.ttf', name='STIXSizeFourSym', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralBol.ttf', name='STIXGeneral', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono.ttf', name='DejaVu Sans Mono', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-Oblique.ttf', name='DejaVu Sans', style='oblique', variant='normal', weight=400, stretch='normal', size='scalable')) = 1.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono-BoldOblique.ttf', name='DejaVu Sans Mono', style='oblique', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralItalic.ttf', name='STIXGeneral', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/cmr10.ttf', name='cmr10', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif.ttf', name='DejaVu Serif', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf', name='DejaVu Sans', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 0.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizTwoSymReg.ttf', name='STIXSizeTwoSym', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneral.ttf', name='STIXGeneral', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-BoldItalic.ttf', name='DejaVu Serif', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSerif-Bold.ttf', name='DejaVu Serif', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-BoldOblique.ttf', name='DejaVu Sans', style='oblique', variant='normal', weight=700, stretch='normal', size='scalable')) = 1.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-Bold.ttf', name='DejaVu Sans', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 0.33499999999999996 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Regular.ttf', name='Liberation Sans Narrow', style='normal', variant='normal', weight=400, stretch='condensed', size='scalable')) = 10.25 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationMono-Italic.ttf', name='Liberation Mono', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSansNarrow-BoldItalic.ttf', name='Liberation Sans Narrow', style='italic', variant='normal', weight=700, stretch='condensed', size='scalable')) = 11.535 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Bold.ttf', name='Liberation Sans Narrow', style='normal', variant='normal', weight=700, stretch='condensed', size='scalable')) = 10.535 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationMono-Bold.ttf', name='Liberation Mono', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSans-Italic.ttf', name='Liberation Sans', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSerif-Italic.ttf', name='Liberation Serif', style='italic', variant='normal', weight=400, stretch='normal', size='scalable')) = 11.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationMono-BoldItalic.ttf', name='Liberation Mono', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf', name='Liberation Sans', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Italic.ttf', name='Liberation Sans Narrow', style='italic', variant='normal', weight=400, stretch='condensed', size='scalable')) = 11.25 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSerif-BoldItalic.ttf', name='Liberation Serif', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSerif-Bold.ttf', name='Liberation Serif', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSans-BoldItalic.ttf', name='Liberation Sans', style='italic', variant='normal', weight=700, stretch='normal', size='scalable')) = 11.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationMono-Regular.ttf', name='Liberation Mono', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSerif-Regular.ttf', name='Liberation Serif', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/humor-sans/Humor-Sans.ttf', name='Humor Sans', style='normal', variant='normal', weight=400, stretch='normal', size='scalable')) = 10.05 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: score(FontEntry(fname='/usr/share/fonts/truetype/liberation/LiberationSans-Bold.ttf', name='Liberation Sans', style='normal', variant='normal', weight=700, stretch='normal', size='scalable')) = 10.335 [2023-05-13 17:01:16] DEBUG - matplotlib.font_manager - findfont: Matching sans\-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0 to DejaVu Sans ('/usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000. [2023-05-13 17:03:47] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 17:03:48] WARNING - super_gradients.training.datasets.detection_datasets.detection_dataset - Found 1090 invalid bbox that were ignored. For more information, please set `show_all_warnings=True`. [2023-05-13 17:03:48] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 17:03:49] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 17:03:49] WARNING - super_gradients.training.datasets.detection_datasets.detection_dataset - Found 4 invalid bbox that were ignored. For more information, please set `show_all_warnings=True`. [2023-05-13 17:04:34] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 17:04:36] INFO - super_gradients.training.utils.checkpoint_utils - License Notification: YOLO-NAS pre-trained weights are subjected to the specific license terms and conditions detailed in https://github.com/Deci-AI/super-gradients/blob/master/LICENSE.YOLONAS.md By downloading the pre-trained weight files you agree to comply with these terms. [2023-05-13 17:05:36] INFO - super_gradients.training.sg_trainer.sg_trainer - Using EMA with params {'decay': 0.9, 'decay_type': 'threshold'} [2023-05-13 17:05:40] DEBUG - tensorflow - Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client. [2023-05-13 17:05:40] DEBUG - h5py._conv - Creating converter from 7 to 5 [2023-05-13 17:05:40] DEBUG - h5py._conv - Creating converter from 5 to 7 [2023-05-13 17:05:40] DEBUG - h5py._conv - Creating converter from 7 to 5 [2023-05-13 17:05:40] DEBUG - h5py._conv - Creating converter from 5 to 7 [2023-05-13 17:05:41] DEBUG - jaxlib.mlir._mlir_libs - Initializing MLIR with module: _site_initialize_0 [2023-05-13 17:05:41] DEBUG - jaxlib.mlir._mlir_libs - Registering dialects from initializer <module 'jaxlib.mlir._mlir_libs._site_initialize_0' from '/usr/local/lib/python3.10/dist-packages/jaxlib/mlir/_mlir_libs/_site_initialize_0.so'> [2023-05-13 17:05:41] DEBUG - jax._src.path - etils.epath found. Using etils.epath for file I/O. [2023-05-13 17:05:42] INFO - numexpr.utils - NumExpr defaulting to 2 threads. [2023-05-13 17:05:49] INFO - super_gradients.training.utils.sg_trainer_utils - TRAINING PARAMETERS: - Mode: Single GPU - Number of GPUs: 1 (1 available on the machine) - Dataset size: 6086 (len(train_set)) - Batch size per GPU: 16 (batch_size) - Batch Accumulate: 1 (batch_accumulate) - Total batch size: 16 (num_gpus * batch_size) - Effective Batch size: 16 (num_gpus * batch_size * batch_accumulate) - Iterations per epoch: 380 (len(train_loader)) - Gradient updates per epoch: 380 (len(train_loader) / batch_accumulate) [2023-05-13 17:13:26] INFO - super_gradients.training.sg_trainer.sg_trainer - [MODEL TRAINING EXECUTION HAS BEEN INTERRUPTED]... Please wait until SOFT-TERMINATION process finishes and saves all of the Model Checkpoints and log files before terminating... [2023-05-13 17:13:26] INFO - super_gradients.training.sg_trainer.sg_trainer - For HARD Termination - Stop the process again [2023-05-13 17:13:26] INFO - super_gradients.common.sg_loggers.base_sg_logger - [CLEANUP] - Successfully stopped system monitoring process [2023-05-13 17:13:27] DEBUG - hydra.core.utils - Setting JobRuntime:name=app [2023-05-13 17:13:57] INFO - super_gradients.training.sg_trainer.sg_trainer - Using EMA with params {'decay': 0.9, 'decay_type': 'threshold'} [2023-05-13 17:13:58] DEBUG - super_gradients.training.utils.sg_trainer_utils - "events.out.tfevents.1683998007.cf6772931124.3382.1" will not be deleted [2023-05-13 17:13:58] DEBUG - super_gradients.training.utils.sg_trainer_utils - "events.out.tfevents.1683997543.cf6772931124.3382.0" will not be deleted |