| Ticket Name: TDA2EXEVM: Assertion @ Line: 232 in dupLink_tsk.c: status == SYSTEM_LINK_STATUS_SOK : failed !! | |
| Query Text: | |
| Part Number: TDA2EXEVM Hi TI Forum, We are trying to run below use-case on our custom board having TDA2xx SoC. We are using Vision SDK 3.0.7. Use-case.txt UseCase: chains_6CH_vdec_encode Capture -> Dup -> VPE -> Encode -> IPCIn(A15) -> Link writes data to shared memory Dup -> Dup_1 -> VPE_D1 -> Merge -> Display Dup_1 -> Sync -> Alg_DmaSwMs -> VPE_D2 ->Merge -> Display sometimes i am facing an assert mentioned below, [HOST] [IPU2 ] 43.816141 s: CAPTURE: Start Done !!! [HOST] [IPU2 ] 43.944702 s: Assertion @ Line: 232 in dupLink_tsk.c: status == SYSTEM_LINK_STATUS_SOK : failed !!! Other than this six channel switching and single channel switching works fine. Any help would be highly appreciable. Regards Balaji T | |
| Responses: | |
| Hi, Whats the use of 2 dup links. You can use the same dup link to create 3 output. Capture -> Dup -> VPE -> Encode -> IPCIn(A15) -> Link writes data to shared memory Dup -> VPE_D1 -> Merge -> Display Dup -> Sync -> Alg_DmaSwMs -> VPE_D2 ->Merge -> Display Regards, Anuj | |
| Anuj, In file VisionSDK_SW_Architecture_Details.pdf, it is mentioned that maximum numbers of channels is 6. In my use-case in total i need 18 output channels from capture, thats why i have used two dup links It is possible to duplicate 6 channels three times using single Dup link..? Rgds Balaji T | |
| Hi, You need 6 channels in 3 different queue so that should be possible using a single dup link. Pleased give it a try. Regards, Anuj | |
| Hi Anuj, Thanks for your suggestion, I have used the single Dup for the three outputs. Below is the use-case, Capture -> Dup -> VPE -> Encode -> IPCIn(A15) -> (link that copies frames to shared memory) Dup -> Sync -> Alg_DmaSwMs -> VPE_D2 ->Merge -> Display Dup -> VPE_D1 -> Merge -> Display But it is observed that sometimes, assertion happens in dup link. [HOST] [IPU2 ] 208.850362 s: Assertion @ Line: 233 in dupLink_tsk.c: status == SYSTEM_LINK_STATUS_SOK : failed !!! And also i have attached the log file for reference. Single_dup_assert.txt Rgds, Balaji T | |
| Anuj, Is there any update on this..?? Rgds, Balaji T | |
| Hi, CAn you put a log in DupLink_drvProcessData and check in which call this assert is hit. Print also the outId for this and also check the content of pBuf. Is it in first call or later. Regards, Anuj | |
| Hi, Its been long since any update on this thread. I hope the issue is resolved. If not then please reply below or create a new thread for a different kind of query. Regards, Anuj | |
| Hi Anuj, The assert is coming in the dup output is connected to Dup -> Sync -> Alg_DmaSwMs -> VPE_D2 -> Merge -> Display. I have tried connecting Alg_DmaSwMs to Dup[0], Dup[1] and Dup [2], in all the cases its gives assert. [HOST] [IPU2 ] 92.867412 s: Bj pBuf 99c3f818 && outId 1 [HOST] [IPU2 ] 92.867504 s: Bj pBuf 99c40518 && outId 2 [HOST] [IPU2 ] 92.873269 s: Bj pBuf 99c3de80 && outId 0 [HOST] [IPU2 ] 92.873421 s: Bj pBuf 0 && outId 1 [HOST] [IPU2 ] 92.873482 s: Assertion @ Line: 234 in dupLink_tsk.c: status == SYSTEM_LINK_STATUS_SOK : failed !!! [HOST] [HOST ] 92.740437 s: Def VSDK BufLink: SYSTEM_CMD_START...!!! In priv.c file, i have set the below two values for sync link, (pObj->SyncPrm).syncThreshold = 33; (pObj->SyncPrm).syncDelta = 33; Please let me know is there any things need to change or add. Rgds, Balaji T | |
| Anuj, Is it possible to use Alg_DmaSwMs without sync link..? I have tried the same, Alg_DmaSwMs link without sync link(Usecase shown below). There is no display on the screen. Capture -> Dup -> Alg_DmaSwMs -> VPE_D2 -> Merge -> Display Please suggest me to resolve this assert. Rgds, Balaji T | |
| Hi Anuj, Is there any update on this? Rgds, Balaji | |
| Hi, You have to do some debugging. As you know the issue is dup link is not getting enough system buffer from its output queue. Can you check how many buffer is been assigned and how much free. Where all the used buffer is? Regards, Anuj | |
| Anuj, Dup link has 32 system buffers for every channel which comes in. I tried increasing it to 48 but still we are seeing the Assert issue. In my observation sync link is the one creating the issue, I have tried changing values if threshold and delta in use-case. I set the values of threshold = 16 and delta = 16, where recording and single channel display are fine. when i switch to mosaic display it freezes. My doubt is, in sync link threshold/delta values are related in any way? Rgds, Balajt T | |
| Anuj, Is there any update on this? Rgds, Balaji T | |
| Hi Balaji, as Anuj is out of office I will ping other VisionSDK experts to help, they will reply here soon. Regards, Yordan | |
| Hi Balaji, Alg_DmaSwMs Link expects the input buffer to be "SYSTEM_BUFFER_TYPE_VIDEO_FRAME_CONTAINER" buffer type, which means the input is buffer array, not single buffer. SYNC Link will collect single buffers from each input channel in the specified time window and make them a buffer array to pass to Alg_DmaSwMs Link. So, you need to have SYNC Link before Alg_DmaSwMs Link. For SYNC link configuration, you have to determine the time window based on input frame rate. Do you mean you have 18 output channels from Capture Link? Dup Link will try to duplicate all 18 channels, not 6 each queue. If you have 18 output channels from Capture Link, you can use Select Link for Mux, 6 for each output queue. Regards, Stanley | |
| Stanley, Thanks for the reply. To determine time window based on input frame rate, is it related to sync threshold and delta values .? If not how to calculate time window based on input frame rate. Do you mean you have 18 output channels from Capture Link? Dup Link will try to duplicate all 18 channels, not 6 each queue. No, I have only 6 channels from capture link. In-fact thats is what i am giving it to single view display. We also kept a select link as shown, Capture -> Dup -> VPE -> Encode -> IPCIn(A15) -> (Link that writes to shared memory) Dup -> Select -> Sync -> Alg_DmaSwMs -> VPE_D2 -> Merge -> Display Dup -> VPE_D1 -> Merge -> Display And set the select link params as numOutQue = 1, outQueId = 0, numOutCh = 6. Also attached the log file for your reference. dup_assert_balaji.txt Needed more support to resolve this issue soon. Thanks & Rgds, Balaji T | |
| The Select Link is redundant in your use case. I mentioned Select Link only because I thought the muxing of input channels were needed in your use case. Based on the Assertion, you should check the error in ~\vision_sdk\links_fw\src\rtos\links_ipu\display\displayLink_drv.c @ line 232. [HOST] [IPU2 ] 60.023638 s: Assertion @ Line: 232 in dupLink_tsk.c: status == SYSTEM_LINK_STATUS_SOK : failed !!! My guess is Dup Link is running out the output frame buffers because the Links after Dup Link is not returning the buffer in time. You have to debug your use case to find out why. Maybe try just 2 input channels first, instead of 6. For Sync Link configuration, please refer to API guide, ~/vision_sdk/docs/VisionSDK_ApiGuide.CHM. You can also take a look at the use case under ~\vision_sdk\apps\src\rtos\usecases\lvds_vip_multi_cam_view for reference. Regards, Stanley | |
| Hi Stanley, Thanks for the suggestion. I have tried with 5 channels, there is no asserts. validated for multiple times. With the help of printing outId in DupLink_drvProcessData function. one can tell that sync link is running slow, i.e., not releasing buffers on time. Will you please let me know how to increase output frame buffers in the dup link. Rgds, Balaji T | |
| The issue is probably not related to Dup Link, which is just passing along the buffers to the next link. Dup link only outputs low frame rate because the links after it return buffers in low rate. This could be the SYNC link is dropping frames due to the configurations. Please review your SYNC link configuration based on the API documentation. | |
| Hi Stanley, I have checked the configuration for sync link in lvds_vip_multi_cam use-cases. I have noticed that they are changing sync threshold(Threshold after which buffers from local queue can be dropped, in msecs ) and delta(Delta on which to sync frames, in msec's ) values. I have validated with both tight sync and loose sync configuration both are resulting in same asserts. Rgds, Balaji | |
| Hi Balaji, Please also check the rest of Links in the chain after DUP link and the frame is processed and returned correctly by each Link. Regards, Stanley | |
| Stanley, The below is our usecase, Capture -> Dup -> VPE -> Encode -> IPCIn(A15) -> (link that writes frames to shared memory) Dup -> Sync -> Alg_DmaSwMs -> VPE_D2 -> Merge -> Display Dup -> VPE_D1 -> Merge -> Display Since we are capturing six channels at 50fps. Alg_SwMosaic will accecpt only Video Composite buffer as input, we need to group frames coming from Capture link(Here we care using Dup link to duplicate data, that just send same input buffer received from prev link to multiple consumers). To group data from multiple channels sync link is used. Sync link will try to synchronize the frame, whose time stamp is within sync delta. Now if the frame from cam0 goes through different chain than the frame from cam1, we need to include latency for the cam0 also in the sync delta. So lets say capture takes around 30ms to release new data, the delta between cam0 and cam1 frame could be around 63ms. If the two frame's timestamp are within 63ms, they will be merged and sent out as a composite frame. But if that does not match, it will wait for the threshold amount of time to synchronize the frame. if it does not receive frame whose timestamp is within sync delta within threshold time, then it will drop the frame. Following is the observation: When we configured the capture link for 5 channels, and sync delta=16(Delta on which to sync frames, in msec's) & threshold=33 (Threshold after which buffers from local queue can be dropped, in msecs) processing of frames are fine and no asserts. Reference log attached(5ch_nov5_debug.txt). i.e., Number of channels is low and processing frames is fine.5ch_nov5_debug.txt When we configured the capture link for 6 channels, and sync delta=16(Delta on which to sync frames, in msec's) & threshold=33 (Threshold after which buffers from local queue can be dropped, in msecs) processing of frames are not proper hence sync link is not making the empty buffer availability to prev link(Dup), which resuts in asserts. Reference log attached(5ch_nov5_debug.txt). We have also tried configuring sync delta and threshold to low (~10) In that interval every frame it getting dropped, no frames is being sent to next link in the chain. So display will be empty. Reference log attached(6ch_nov5_debug.txt)6ch_nov5_debug.txt Rgds, Balaji | |
| For 50 fps inputs, the sync threshold is (1/50) * 1000 = 20 ms. It should expect new frame from each input channel in 20ms interval. As for delta, you can probably start with half of the interval, which is 10. | |
| In addition to that, you should increase the number of max elements in queue. 10 is good for 2 frames in 5 channels. But you need at least 12 for 2 frames in 6 channels. Find below under ~/vision_sdk/links_fw/src/rtos/links_common/sync/syncLink_priv.h /** ******************************************************************************* * \brief Max number of elemtns in sync link local queue ******************************************************************************* */ #define SYNC_LINK_LOCAL_QUE_MAX_ELEMENTS (10U) /** ******************************************************************************* * * \brief Channel specific information * ******************************************************************************* */ typedef struct { Utils_QueHandle localQueHandle; /**< Handle to the queue for this channel */ System_Buffer *queMem[SYNC_LINK_LOCAL_QUE_MAX_ELEMENTS]; /**< Queue memory */ } SyncLink_ChObj; | |
| stanley, why sometimes, sync link is dropping half of the frames its receiving. The below is my use-case(rearranged some elements) Capture -> VPE -> Dup -> Encode -> IPCIn(A15) -> DefLink_vsdkbuf (A15) Dup -> Merge -> Display Dup -> Sync -> Alg_DmaSwMs -> VPE_D2 -> Merge -> Display I have enabled print statistics for sync link, after vpe link sync link dropping half of the frames. That makes video on the screen slow. Please provide some solution for this, Rgds, Balaji T | |
| Did the assertion go away after the change mentioned previously? As for Sync link dropping frame, could you find out what cause the drops? Is it because incoming frames do not arrive within the threshold specified in create params? | |
| Stanley, Yes, the assertion was gone, but i am not sure that is because increasing max queue elements or rearranging of elements in the usecase. If sync link is dropping frames because of incoming frames do not arrive within the threshold time interval, why it is not dropping every time ? sync link is dropping frames only sometimes. Regards Balaji T | |
| You can add some trace in SyncLink_addBuffersToDropListLesserThanThreshold() SyncLink_addBuffersToDropListFromLocalQue() under ~/vision_sdk/links_fw/src/rtos/links_common/sync/syncLink_tsk.c to see why buffers are dropped. Regards, Stanley | |
| Balaji Were you able to proceed on this one? Thanks and Regards Piyali | |
| Hi Piyali, As you can see all the above comments, please correct me if i misunderstood any concepts. yes, still i am facing the same issue at-least twice in 10 times. We have validating the same, please have a look in the log.6ch_assert_validation.txt Detailed log with VSDK stats collector s given below. FS-6ch_debug_nov28.txt Regards & Thanks, Balaji T | |
| Hi, Is there any update on this..? Rgds, Balaji T | |
| Did you add any trace as mentioned earlier in the below functions? SyncLink_addBuffersToDropListLesserThanThreshold() SyncLink_addBuffersToDropListFromLocalQue() under ~/vision_sdk/links_fw/src/rtos/links_common/sync/syncLink_tsk.c to see why buffers are dropped. I didn't find anything useful from the log you shared. Regards, Stanley | |
| Please let us know if you are still facing any issue. Thanks. | |
| Hi Stanley, The dup link assert and the frame drop in sync link is not happening all the times. In the above shared logs, the first one is to analyze how many times the assert in dup link is occurred and the second long is with enabled all the print statistics i.e., the log generated by pressing 'p' . We added trace logs in the follow functions but we dint find any reference that makes frame drop, SyncLink_addBuffersToDropListLesserThanThreshold() SyncLink_addBuffersToDropListFromLocalQue() Regards, Balaji T | |
| Hi Balaji, Could you try increasing the value of SYNC_LINK_LOCAL_QUE_MAX_ELEMENTS to see if it eliminates the assertion in Dup Link? You can try a large number like 24 from 10. /** ******************************************************************************* * \brief Max number of elemtns in sync link local queue ******************************************************************************* */ #define SYNC_LINK_LOCAL_QUE_MAX_ELEMENTS (10U) Regards, Stanley | |
| Hi Balaji, Please let me know if the suggested change makes any difference. Thanks. Regards, Stanley | |
| Stanley, I tried gradually increasing the value of SYNC_LINK_LOCAL_QUE_MAX_ELEMENTS from 12 to 24, sometimes once in 5 times getting assert. Rgds, Balaji T | |
| Hi Balaji, If you are seeing assertion, the trace should tell you where the assertion is. Could you then trace the error back to what inside the function that returned bad status to cause the assertion? I need to know what exactly the condition check fails which triggers the assertion at the end and then we could try to figure out the root cause. So far, the information you have given is only the first level function which return bad status but we need further detail into that function and the condition check which failed. Regards, Stanley | |
| Hi Balaji, Any update on this issue? Thanks. Regards, Stanley | |