File size: 18,025 Bytes
5aefcf4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 |
Ticket Name: TDA2: New model runs well in tidl_model_import.out.exe but abnormally in CCS and EVM Query Text: Part Number: TDA2 Hi, I train a new model, and runs well in in tidl_model_import.out.exe, and the result is shown in attachment. Then I run the model in EVM TIDL_OD usecase, it comes out the unexpected result.(show in attachment). At last, I run the model in CCS, and also got unexpected result. I check the trace_dump files between CCS and tidl_model_import.out.exe, and all layers are unmatched except input data layer. So could you please help to check my issue. and if prototxt and caffemodel is needed, please provide email. issueTIDL.rar Thanks Jerry Responses: Hi Jerry, What is the TIDL release version you are using? Please share the all the dumped outputs (traces) of import and CCS for checking the issue Thanks, Praveen Hi Praveen, TIDL: REL.TIDL.01.01.01.00 CCS: v8.1 VISIONSDK: 3.04 Thanks, Jerry 6131.issueTIDL.rar Hi Praveen, I try simulator my model in PC with eve_test_dl_algo.exe, and get the following results. 1 DenseConv may be failed when output channels less than 48, or stride=2, or dilate=2 as TIDL has these limits 2 But why sparseConv failed at the following situations? PC_simulator.log Layer 1 : Out Q : 12122 , TIDL_BatchNormLayer , PASSED #MMACs = 0.50, 0.50, Sparsity : 0.00 Layer 2 : Out Q : 16454 , TIDL_BatchNormLayer , PASSED #MMACs = 0.50, 0.50, Sparsity : 0.00 Layer 3 : Out Q : 44315 , TIDL_ConvolutionLayer, PASSED #MMACs = 35.83, 30.19, Sparsity : 15.74 Layer 4 : Out Q : 29847 , TIDL_ConvolutionLayer, PASSED #MMACs = 42.47, 42.47, Sparsity : 0.00 Layer 5 : Out Q : 18574 , TIDL_ConvolutionLayer, PASSED #MMACs = 11.94, 11.94, Sparsity : 0.00 Layer 6 : Out Q : 10320 , TIDL_ConvolutionLayer, PASSED #MMACs = 84.93, 84.93, Sparsity : 0.00 Layer 7 : Out Q : 20910 , TIDL_ConvolutionLayer, PASSED #MMACs = 23.89, 23.89, Sparsity : 0.00 Layer 8 :TIDL_PoolingLayer, PASSED #MMACs = 0.66, 0.66, Sparsity : 0.00 Layer 9 : Out Q : 30028 , TIDL_ConvolutionLayer, PASSED #MMACs = 21.23, 21.23, Sparsity : 0.00 Layer 10 : Out Q : 33360 , TIDL_ConvolutionLayer, PASSED #MMACs = 2.99, 2.99, Sparsity : 0.00 Layer 11 : Out Q : 20992 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 12 : Out Q : 30337 , TIDL_ConvolutionLayer, PASSED #MMACs = 31.85, 31.85, Sparsity : 0.00 Layer 13 : Out Q : 29768 , TIDL_ConvolutionLayer, PASSED #MMACs = 2.99, 2.99, Sparsity : 0.00 Layer 14 : Out Q : 21075 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 15 : Out Q : 30007 , TIDL_ConvolutionLayer, PASSED #MMACs = 42.47, 42.47, Sparsity : 0.00 Layer 16 : Out Q : 29477 , TIDL_ConvolutionLayer, PASSED #MMACs = 2.99, 2.99, Sparsity : 0.00 Layer 17 : Out Q : 21158 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 18 : Out Q : 61360 , TIDL_ConvolutionLayer, PASSED #MMACs = 212.34, 212.34, Sparsity : 0.00 Layer 19 :TIDL_PoolingLayer, PASSED #MMACs = 0.33, 0.33, Sparsity : 0.00 Layer 20 : Out Q : 39417 , TIDL_ConvolutionLayer, PASSED #MMACs = 15.93, 15.93, Sparsity : 0.00 Layer 21 : Out Q : 46384 , TIDL_ConvolutionLayer, PASSED #MMACs = 1.12, 1.12, Sparsity : 0.00 Layer 22 : Out Q : 46567 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 23 : Out Q : 42536 , TIDL_ConvolutionLayer, PASSED #MMACs = 21.90, 21.90, Sparsity : 0.00 Layer 24 : Out Q : 44216 , TIDL_ConvolutionLayer, PASSED #MMACs = 1.12, 1.12, Sparsity : 0.00 Layer 25 : Out Q : 44390 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 26 : Out Q : 40258 , TIDL_ConvolutionLayer, PASSED #MMACs = 27.87, 27.87, Sparsity : 0.00 Layer 27 : Out Q : 52020 , TIDL_ConvolutionLayer, PASSED #MMACs = 1.12, 1.12, Sparsity : 0.00 Layer 28 : Out Q : 44565 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 29 : Out Q : 41968 , TIDL_ConvolutionLayer, PASSED #MMACs = 33.84, 33.84, Sparsity : 0.00 Layer 30 : Out Q : 32649 , TIDL_ConvolutionLayer, PASSED #MMACs = 1.12, 1.12, Sparsity : 0.00 Layer 31 : Out Q : 32778 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 32 : Out Q : 41851 , TIDL_ConvolutionLayer, PASSED #MMACs = 39.81, 39.81, Sparsity : 0.00 Layer 33 : Out Q : 41670 , TIDL_ConvolutionLayer, PASSED #MMACs = 1.12, 1.12, Sparsity : 0.00 Layer 34 : Out Q : 32907 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 35 : Out Q : 55923 , TIDL_ConvolutionLayer, PASSED #MMACs = 122.09, 122.09, Sparsity : 0.00 Layer 36 :TIDL_PoolingLayer, PASSED #MMACs = 0.08, 0.08, Sparsity : 0.00 Layer 37 : Out Q : 33730 , TIDL_ConvolutionLayer, PASSED #MMACs = 5.31, 5.31, Sparsity : 0.00 Layer 38 : Out Q : 51053 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.37, 0.37, Sparsity : 0.00 Layer 39 : Out Q : 51254 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 40 : Out Q : 39601 , TIDL_ConvolutionLayer, PASSED #MMACs = 7.96, 7.96, Sparsity : 0.00 Layer 41 : Out Q : 50233 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.37, 0.37, Sparsity : 0.00 Layer 42 : Out Q : 50431 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 43 : Out Q : 29201 , TIDL_ConvolutionLayer, PASSED #MMACs = 10.62, 10.62, Sparsity : 0.00 Layer 44 : Out Q : 35166 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.37, 0.37, Sparsity : 0.00 Layer 45 : Out Q : 35304 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 46 : Out Q : 30263 , TIDL_ConvolutionLayer, PASSED #MMACs = 13.27, 13.27, Sparsity : 0.00 Layer 47 : Out Q : 31795 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.37, 0.37, Sparsity : 0.00 Layer 48 : Out Q : 31920 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 49 : Out Q : 29030 , TIDL_ConvolutionLayer, PASSED #MMACs = 15.93, 15.93, Sparsity : 0.00 Layer 50 : Out Q : 34302 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.37, 0.37, Sparsity : 0.00 Layer 51 : Out Q : 32046 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 52 : Out Q : 31669 , TIDL_ConvolutionLayer, PASSED #MMACs = 18.58, 18.58, Sparsity : 0.00 Layer 53 : Out Q : 33290 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.37, 0.37, Sparsity : 0.00 Layer 54 : Out Q : 32172 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 55 : Out Q : 37877 , TIDL_ConvolutionLayer, PASSED #MMACs = 42.47, 42.47, Sparsity : 0.00 Layer 56 : Out Q : 30049 , TIDL_ConvolutionLayer, PASSED #MMACs = 7.96, 7.96, Sparsity : 0.00 Layer 57 : Out Q : 40896 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.56, 0.56, Sparsity : 0.00 Layer 58 : Out Q : 38026 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 59 : Out Q : 23376 , TIDL_ConvolutionLayer, PASSED #MMACs = 13.93, 13.93, Sparsity : 0.00 Layer 60 : Out Q : 40974 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.56, 0.56, Sparsity : 0.00 Layer 61 : Out Q : 38176 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 62 : Out Q : 25187 , TIDL_ConvolutionLayer, PASSED #MMACs = 19.91, 19.91, Sparsity : 0.00 Layer 63 : Out Q : 34145 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.56, 0.56, Sparsity : 0.00 Layer 64 : Out Q : 34279 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 65 : Out Q : 29118 , TIDL_ConvolutionLayer, PASSED #MMACs = 25.88, 25.88, Sparsity : 0.00 Layer 66 : Out Q : 35737 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.56, 0.56, Sparsity : 0.00 Layer 67 : Out Q : 34414 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 68 : Out Q : 14894 , TIDL_ConvolutionLayer, PASSED #MMACs = 42.47, 42.47, Sparsity : 0.00 Layer 69 :TIDL_PoolingLayer, PASSED #MMACs = 0.08, 0.08, Sparsity : 0.00 Layer 70 : Out Q : 27848 , TIDL_ConvolutionLayer, PASSED #MMACs = 10.62, 10.62, Sparsity : 0.00 Layer 71 : Out Q : 14953 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 72 : Out Q : 23194 , TIDL_ConvolutionLayer, PASSED #MMACs = 10.62, 10.62, Sparsity : 0.00 Layer 73 : Out Q : 44331 , Failing at 0, 0, 0, 0 ref,out = 1,255 TIDL_ConvolutionLayer, FAILED!!!!!! #MMACs = 0.19, 0.19, Sparsity : 0.00 Layer 74 : Out Q : 18410 , TIDL_ConvolutionLayer, PASSED #MMACs = 2.65, 2.65, Sparsity : 0.00 Layer 75 :TIDL_PoolingLayer, PASSED #MMACs = 0.04, 0.04, Sparsity : 0.00 Layer 76 : Out Q : 13003 , TIDL_ConvolutionLayer, PASSED #MMACs = 5.31, 5.31, Sparsity : 0.00 Layer 77 : Out Q : 13054 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 78 : Out Q : 21503 , TIDL_ConvolutionLayer, PASSED #MMACs = 2.65, 2.65, Sparsity : 0.00 Layer 79 : Out Q : 52782 , Failing at 0, 0, 0, 4 ref,out = 26,0 TIDL_ConvolutionLayer, FAILED!!!!!! #MMACs = 0.05, 0.05, Sparsity : 0.00 Layer 80 : Out Q : 9095 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.74, 0.74, Sparsity : 0.00 Layer 81 :TIDL_PoolingLayer, PASSED #MMACs = 0.01, 0.01, Sparsity : 0.00 Layer 82 : Out Q : 9817 , TIDL_ConvolutionLayer, PASSED #MMACs = 1.47, 1.47, Sparsity : 0.00 Layer 83 : Out Q : 9131 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 84 : Out Q : 37538 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.74, 0.74, Sparsity : 0.00 Layer 85 : Out Q : 87882 , Failing at 0, 0, 0, 1 ref,out = 0,255 TIDL_ConvolutionLayer, FAILED!!!!!! #MMACs = 0.02, 0.02, Sparsity : 0.00 Layer 86 : Out Q : 29973 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.25, 0.25, Sparsity : 0.00 Layer 87 :TIDL_PoolingLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 88 : Out Q : 16981 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.49, 0.49, Sparsity : 0.00 Layer 89 : Out Q : 17048 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 90 : Out Q : 65965 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.25, 0.25, Sparsity : 0.00 Layer 91 : Out Q : 111900 , Failing at 0, 0, 0, 0 ref,out = 47,255 TIDL_ConvolutionLayer, FAILED!!!!!! #MMACs = 0.01, 0.01, Sparsity : 0.00 Layer 92 : Out Q : 53030 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.10, 0.10, Sparsity : 0.00 Layer 93 :TIDL_PoolingLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 94 : Out Q : 44532 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.20, 0.20, Sparsity : 0.00 Layer 95 : Out Q : 44707 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 96 : Out Q : 86672 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.10, 0.10, Sparsity : 0.00 Layer 97 : Out Q : 130284 , Failing at 0, 0, 0, 0 ref,out = 40,255 TIDL_ConvolutionLayer, FAILED!!!!!! #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 98 : Out Q : 80025 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.03, 0.03, Sparsity : 0.00 Layer 99 :TIDL_PoolingLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 100 : Out Q : 56297 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.07, 0.07, Sparsity : 0.00 Layer 101 : Out Q : 56519 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 102 : Out Q : 7137 , TIDL_ConvolutionLayer, PASSED #MMACs = 2.65, 2.65, Sparsity : 0.00 Layer 103 : Out Q : 3821 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.09, 0.09, Sparsity : 0.00 Layer 104 :TIDL_FlattenLayer, PASSED #MMACs = 0.01, 0.01, Sparsity : 0.00 Layer 105 : Out Q : 10029 , TIDL_ConvolutionLayer, PASSED #MMACs = 2.65, 2.65, Sparsity : 0.00 Layer 106 : Out Q : 5602 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.09, 0.09, Sparsity : 0.00 Layer 107 :TIDL_FlattenLayer, PASSED #MMACs = 0.01, 0.01, Sparsity : 0.00 Layer 108 : Out Q : 7778 , TIDL_ConvolutionLayer, PASSED #MMACs = 1.00, 1.00, Sparsity : 0.00 Layer 109 : Out Q : 10903 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.03, 0.03, Sparsity : 0.00 Layer 110 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 111 : Out Q : 5431 , TIDL_ConvolutionLayer, PASSED #MMACs = 1.00, 1.00, Sparsity : 0.00 Layer 112 : Out Q : 4815 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.03, 0.03, Sparsity : 0.00 Layer 113 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 114 : Out Q : 7993 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.28, 0.28, Sparsity : 0.00 Layer 115 : Out Q : 10285 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.01, 0.01, Sparsity : 0.00 Layer 116 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 117 : Out Q : 5102 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.28, 0.28, Sparsity : 0.00 Layer 118 : Out Q : 4925 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.01, 0.01, Sparsity : 0.00 Layer 119 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 120 : Out Q : 9731 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.09, 0.09, Sparsity : 0.00 Layer 121 : Out Q : 10006 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 122 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 123 : Out Q : 7466 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.09, 0.09, Sparsity : 0.00 Layer 124 : Out Q : 6931 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 125 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 126 : Out Q : 16688 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.04, 0.04, Sparsity : 0.00 Layer 127 : Out Q : 14009 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 128 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 129 : Out Q : 12677 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.04, 0.04, Sparsity : 0.00 Layer 130 : Out Q : 8489 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 131 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 132 : Out Q : 33500 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.01, 0.01, Sparsity : 0.00 Layer 133 : Out Q : 17943 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 134 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 135 : Out Q : 12216 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.01, 0.01, Sparsity : 0.00 Layer 136 : Out Q : 7388 , TIDL_ConvolutionLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 137 :TIDL_FlattenLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : 0.00 Layer 138 : Out Q : 3836 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 139 : Out Q : 4834 , TIDL_ConcatLayer, PASSED #MMACs = 0.00, 0.00, Sparsity : -nan(ind) Layer 140 : #MMACs = 0.00, 0.00, Sparsity : 0.00 End of config list found ! Hi Jerry, For these conv layers, the number of groups are equal to number of output channels and for this case we had a separate flow(depth wise separable conv flow) and in this flow there is no support for dilation >1, so for these conv layers you can have dilation = 1 or decrease number of groups so that it will not take this flow. Thanks, Praveen Hi Praveen, 1. In depthwise separable flow, both stride>1 and dilation>1 are not supported, right? 2. If I want to stride>1 and dilation>1 are supported in my network, the conv layer must run in sparse conv flow(conv groups != output channels and convKernelType=0). Is it riht? Thanks for your help. Best regards, Jerry Hi Jerry, 1. No, only dilation>1 is not supported, but stride = 1 or 2 is supported. 2. Yes, you are right. Thanks, Praveen Hi Praveen, I will try it after the long vocation. Thank you very much. Best regards, Jerry Hi Praveen, I have tried the following cases: 1. group = num_out, stride=1, dilation=1, conv2dKernelType=0 => fail 2. group = num_out, stride=1, dilation=1, conv2dKernelType=1 => pass 3. group = num_out, stride=2, dilation=1, conv2dKernelType=0 => fail 4. group = num_out, stride=2, dilation=1, conv2dKernelType=1 => fail So If conv layer works well in depthwise separable flow => group=num_out, stride=1, conv2dKernelType=1 ? Best regards, Jerry Hi Jerry, Yes. Thanks, Praveen |