| Ticket Name: Linux/TDA2PXEVM: PCIe Communication between PC and TDA2P EVA board | |
| Query Text: | |
| Part Number: TDA2PXEVM Other Parts Discussed in Thread: TDA2 Tool/software: Linux Dear Ti, we are trying to connect TDA2P EVA board to PC (with Linux running) over PCIe. We managed to connect TDA2P EVA board and TDA2XX EVA board over PCIe following your Wiki page. In this case the TDA2P EVA served as PCIe RC and the TDA2XX EVA as PCIe EP. Both boards are running Linux with kernel 4.4.84 .In this scenario, we where able to run the pcitest.sh PCIe test script and everything worked as expected. Now, we are trying to connect TDA2P EVA board with PC over PCIe. We are using the same PCIe cable that we used when connecting two EVA boards as described above. The TDA2P EVA board is running Linux with kernel 4.4.84. The PC is running Ubuntu 16.04.4 LTS with custom kernel (we recompiled the 4.16.7 Linux kernel to enable the PCI Endpoint Test Driver). The problem we are facing now is that the lspci command sometimes shows the TDA2P EVA board and sometimes does not. When the lspci is able to see TDA2P EVA board we get the following output: 00:00.0 Host bridge: Intel Corporation Device 591f (rev 05) 00:02.0 VGA compatible controller: Intel Corporation Device 5912 (rev 04) 00:14.0 USB controller: Intel Corporation Device a2af 00:16.0 Communication controller: Intel Corporation Device a2ba 00:17.0 SATA controller: Intel Corporation Device a282 00:1c.0 PCI bridge: Intel Corporation Device a294 (rev f0) 00:1c.5 PCI bridge: Intel Corporation Device a295 (rev f0) 00:1d.0 PCI bridge: Intel Corporation Device a298 (rev f0) 00:1f.0 ISA bridge: Intel Corporation Device a2c8 00:1f.2 Memory controller: Intel Corporation Device a2a1 00:1f.3 Audio device: Intel Corporation Device a2f0 00:1f.4 SMBus: Intel Corporation Device a2a3 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V 01:00.0 Unassigned class [ff00]: Texas Instruments Device b500 (rev ff) 02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04) When we remove the TDA2P from PCIe bus (through /sys/bus/pci/devices/<TDA2P_PCIE_ID>/remove) and rescan the PCIe bus (trough /sys/bus/pci/rescan) we are not able to see TDA2P with lspci command. The same behavior is observed when we reboot the PC. Sometimes the PC recognizes the TDA2P sometimes not - the behavior is random. In the rare case when the PC recognizes the TDA2P EVA board we have the following dmesg output on PC side: [ 9612.437772] pci 0000:01:00.0: [104c:b500] type 00 class 0xff0000 [ 9612.437833] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x000000ff] [ 9612.437856] pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x000001ff] [ 9612.437882] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000003ff] [ 9612.437898] pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00003fff] [ 9612.438129] pci 0000:01:00.0: supports D1 [ 9612.438135] pci 0000:01:00.0: PME# supported from D0 D1 D3hot [ 9612.508627] pci 0000:01:00.0: BAR 3: assigned [mem 0xc8000000-0xc8003fff] [ 9612.508637] pci 0000:01:00.0: BAR 3: error updating (0xc8000000 != 0xffffffff) [ 9612.508648] pci 0000:01:00.0: BAR 2: assigned [mem 0xc8004000-0xc80043ff] [ 9612.508655] pci 0000:01:00.0: BAR 2: error updating (0xc8004000 != 0xffffffff) [ 9612.508662] pci 0000:01:00.0: BAR 1: assigned [mem 0xc8004400-0xc80045ff] [ 9612.508669] pci 0000:01:00.0: BAR 1: error updating (0xc8004400 != 0xffffffff) [ 9612.508675] pci 0000:01:00.0: BAR 0: assigned [mem 0xc8004600-0xc80046ff] [ 9612.508682] pci 0000:01:00.0: BAR 0: error updating (0xc8004600 != 0xffffffff) [ 9612.508690] pci 0000:02:00.0: PCI bridge to [bus 03] [ 9612.528477] pci-endpoint-test 0000:01:00.0: Refused to change power state, currently in D3 [ 9612.528667] pci-endpoint-test 0000:01:00.0: failed to get MSI interrupts [ 9612.528822] pci-endpoint-test 0000:01:00.0: can't ioremap BAR 4: [??? 0x00000000 flags 0x0] [ 9612.528826] pci-endpoint-test 0000:01:00.0: failed to read BAR4 [ 9612.528833] pci-endpoint-test 0000:01:00.0: can't ioremap BAR 5: [??? 0x00000000 flags 0x0] [ 9612.528836] pci-endpoint-test 0000:01:00.0: failed to read BAR5 We also enabled additional kernel debugging messages on TDA2P and when we power up the board, the dmesg shows the following message: [ 71.223875] dra7-pcie 51000000.pcie_ep: Link-up state change [ 95.154440] dra7-pcie 51000000.pcie_ep: Link Request Reset [ 95.154456] dra7-pcie 51000000.pcie_ep: Link-up state change In contrast, when we connect the TDA2P EVA and TDA2XX EVA boards, on EP (TDA2XX EVA board) side the dmesg shows following: [ 36.516286] dra7-pcie 51000000.pcie_ep: Link-up state change [ 130.663177] dra7-pcie 51000000.pcie_ep: CFG 'Memory Space Enable' change [ 130.663252] dra7-pcie 51000000.pcie_ep: CFG 'Bus Master Enable' change Do you have any ideas or suggestions for us? Could this all be a cable issue caused by some configuration lines/bits? Thank you in advance, Stefan Laza. | |
| Responses: | |
| Hi, You should always boot EP first and configure endpoint device (bind). Once configuration is done boot the RC. TDA2X doesn't support PCIe hot plug. Regards, RK | |
| This is something we are aware off. We have also tried booting EP first, however we get the same results and logs. Do you have any other propositions? Could this be an issue caused by our ASUS motherboard with intel's PCIe controller on it? We also observed that the EP LTSSM is stuck in DETECT.QUIET state. On EP side we used the following command to read the status register of the LTSSM: devmem2 0x51002104 w the above command response is: Read at address 0x51002104 (0xb6fa6104): 0x00000000 Regards, SL. | |
| Hi Stefan, Is this issue still observed? If yes, could you halt the PC at GRUB/boot-loader? While powering on the PC its possible that the TDA2P EVA is reset and thus the no function is bound to the EP-controller. Let the TDA2PX EVA complete its initialization and bind the function to the EPC. All this while let the PC be in boot-loader and then jump to Linux. Can you also remove the reset resistor on the cable? The log 'Link Request Reset' occurs when the link has gone down and comes back up again, which could be caused by the controller on the PC. Removing the reset resistor could prevent this. Regards Shravan | |
| Hi Shravan, When managed avoid this issue (this is only a temporary solution) by setting the EP device to operate as GEN1 PCIe device. The device tree for our EP (TDA2p EVA board) looks as follows: &pcie1_ep { ti,pcie-is-gen1; status = "okay"; }; With the above modification we are able to 'see' the EP device on our host PC with lspci command. Also, we are able to remove the device and rescan the bus with following commands: echo 1 > /sys/bus/pci/devices/<EP_ID>/remove echo 1 > /sys/bus/pci/rescan and every time it is working as expected. There is no need to power up the EP device first and then the host PC - while our host PC is working, we boot the EP device and do a PCIe bus rescan (with the above command) and the EP device is properly recognized by the host PC. As said above, for us this is only a temorary solution, because we need the EP device to be able to operate as a PCIe GEN2 device. The reset resistor does nto exist on our cable. Regards, Stefan. | |
| Hi Stefan, It looks to be a problem on the RC side while enumerating the EP. Has any other PCIe Gen-2 device been tested / can you validate this on another PC? We validate it on PCIe Gen-2 and it works. Regards Shravan | |
| Hi Shravan, Our motherboard on which we are testing is: ASUS B250M-C PRO REV 1.01 Do you have any suggestion which model to use? On this motherboard we tested intel SSD with PCIe interface and the interface is working as expected. Regards, Stefan | |
| Hi Stefan, The issue seems to be with the link being unstable. There was a fix in a subsequent release (in kernel 4.14), you can find the patch here (for K-4.14). You may need to back-port it your kernel. This should resolve the issue with the link stability. git.ti.com/.../phy-ti-pipe3.c Regards Shravan | |
| Dear Shravan, We tried the patch that you recommended, but the issue is still present. When we power up the EP, in the kernel log we have: [ 71.223875] dra7-pcie 51000000.pcie_ep: Link-up state change [ 95.154440] dra7-pcie 51000000.pcie_ep: Link Request Reset [ 95.154456] dra7-pcie 51000000.pcie_ep: Link-up state change And the host PC is not able to see the EP when we invoke lspci. Again, when we set the PCIE on EP to operate as GEN1 in device tree: &pcie1_ep { ti,pcie-is-gen1; status = "okay"; }; on EP side in kernel log we get: [ 41.125133] dra7-pcie 51000000.pcie_ep: Link-up state change [ 52.756835] dra7-pcie 51000000.pcie_ep: CFG 'Memory Space Enable' change [ 52.757052] dra7-pcie 51000000.pcie_ep: CFG 'Bus Master Enable' change and from this point the host PC is able to see the EP. We will try with another motherboard, and we will keep you informed about the results. Regards, Stefan. | |
| Hi, Do you see same behavior when connected to a different PC? Also, Try a different board if you have. Regards, LB | |
| Yes, the same behavior is observed with different motherboards. So far we tried two models of motherboards: 1. ASUS B250-C PRO REV 1.01 2. ASRock Z68M/USB3 We have two EVA boards: TDA2XX EVA TDA2P EVA We tried both of them with the above mentioned motherboards - the behavior is same - we only get the boards recognized by the host computer when they are configured to operate as GEN1 PCIe devices. When we configure the EP side (TDA2P or TDA2XX) to operate as GEN2 PCIe device and when we read the status of register PCIECTRL_TI_CONF_DEVICE_CMD (0x5100 2104) using the devmem2 command, we get: root@dra7xx-evm:~# devmem2 0x51002104 /dev/mem opened. Memory mapped at address 0xb6ff1000. Read at address 0x51002104 (0xb6ff1104): 0x00000000 According to TDA2XX TRM the bits 2-7 in PCIECTRL_TI_CONF_DEVICE_CMD represent the LTSSM_STATE and in our case it seems that the LTSSM ends up in state DETECT_QUIET. The bit 0 PCIECTRL_TI_CONF_DEVICE_CMD indicates that the LTSSM is not enabled. In contrast, when configure our boards to operate as GEN1 devices and read PCIECTRL_TI_CONF_DEVICE_CMD register we get: root@dra7xx-evm:~# devmem2 0x51002104 /dev/mem opened. Memory mapped at address 0xb6ff5000. Read at address 0x51002104 (0xb6ff5104): 0x00000041 Again, according to TDA2XX TRM, the LTSSM is in RCVRY_IDLE. And the LTSSM is enabled. Do you have any idea why is this happening? Regards, Stefan. | |
| In the bellow mentioned scenarios we use the PCIe cable modification as suggested on your wiki page: processors.wiki.ti.com/.../Linux_Core_PCIe_EP_User's_Guide We use x1 PCIe cable. So far we tried following configurations: 1. TDA2XX (EP) (Linux) -> TDA2P(RC) (Linux) - with PCIe Endpoint test drivers the boards are successfully configured as PCIe GEN2 devices and the pcitest.sh script finishes without errors. 2. TDA2P (EP) (Linux) -> TDA2XX(RC) (Linux) - with PCIe Endpoint test drivers the boards are successfully configured as PCIe GEN2 devices and the pcitest.sh script finishes without errors. 3. TDA2XX (EP) (Linux) -> Host PC (Linux) - with PCIe Endpoint test drivers the Host PC is able to recognize the EP only as GEN1 the pcitest.sh test crashes randomly - the EP board freezes on Read, Write and Copy tests. (sometimes even the Host PC freezes) 4. TDA2P (EP) (Linux) -> Host PC (Linux) - with PCIe Endpoint test drivers the Host PC is able to recognize the EP only as GEN1 the pcitest.sh test crashes randomly - the EP board freezes on Read, Write and Copy tests. (sometimes even the Host PC freezes) On our custom boards we have 3 SOC (TDA2XX) - two of them are connected with PCIe on PCB. On this board we are able to run similar test - like 1. and 2. - and the test are done without errors. We also wrote our custom drivers for EP and RC, based on PCIe Endpoint test driver code. In this drivers we implemented DMA transfer from EP to RC device. On our custom board the DMA transfer is done without errors. But when we connect TDA2XX or TDA2P EVA board to the Host PC and try the DMA transfer (we tried to transfer buffers of 4KB in for loop) - the PCIe link is reset during the for loop iteration - it happens at random intervals (sometimes the link is reset after 100 iterations, sometimes after 1000 - there is no pattern). In our custom driver we tried memcpy from EP to Host PC, but the EP freezes randomly during memcpy data transfer. When we try the same memcpy code on our custom board, the data transfer is done without error. Is there any possibility that the Host PC can't recognize the EP as PCIe GEN2 because the reference clock signals are disconnected on the PCIe cable (we disconnected the reference clock signals based on the Wiki page mentioned above)? Could this missing reference clock signals be the reason why memcpy and DMA data transfer is not working between EP (TDA2XX or TDA2PX) and Host PC? Can you tell us exactly how you test the PCIe communication between TDA2(P)XX EVA boards? Do you use Linux on EP and RC or do you only test with ROTS running on both of them? Regards, Stefan. | |
| Hi Stefan, It does look like an issue with the reference clock supplied to the EP. To ensure that the PC provides the reference clock to the EP (over PCIe), please modify the following (on the TDA2XX EVM): 1. Populate the reference clock signal on the cable 2. Depopulate capacitors C120/C125/C259/C262 3. Populate 0-ohm resistors across C120/C259 and C125/C262 (shown in the fig) 4. Depopulate resistors R806/R807 5. Resistor R223 i) Populate with 100ohm if PCIE Connector clock is LVDS ii) Depopulate if PCIE Connector clock is HCSL Also in the device-tree file dra7xx-clocks.dtsi, modify the reg attribute of the node apll_pcie_in_clk_mux as below reg = <0x21x 0x84> Regards Shravan | |
| Hi Shravan, we made the hardware changes that you proposed and so far this instructions are clear. But, we do not understand the change in dra7xx-clock.dtsi file. You proposed to change the following value: reg = <0x021x 0x4>; to reg = <0x021x 0x84>; According to device tree binding documentation found in (kernel_source/Documentation/devicetree/bindings) the second parameter for reg represents the register size. When we boot our EVA board and read the register value for CM_CLKMODE_APLL_PCIE (0x4a00821c), we get the following result: root@dra7xx-evm:~# devmem2 0x4a00821c /dev/mem opened. Memory mapped at address 0xb6f93000. Read at address 0x4A00821C (0xb6f9321c): 0x00000101 According to TDA2XX TRM bit 7 of CM_CLKMODE_APLL_PCIE should be set to 1 in order to select PCIe connector clock to be the reference clock for APLL_PCIE. In our case the bit 7 is 0 which means that the APLL_PCIE is still driven by the internal PLL (DPLL_PCIE_REF). Do you have any other suggestions on how to change the reference clock source for the APLL_PCIE? Regards, Stefan. | |
| Hi Stefan, Please ignore the DT change suggested in the previous post (however please perform the hardware modifications suggested). I've attached a patch which should enable APLL reference input clock from ACSPCIE. Regards Shravan 7875.0001-phy-pcie-Set-APLL-reference-input-clock-as-ACSPCIE.patch.txt From 5a16e6f5223b06e076c5d97f57b626721486a03a Mon Sep 17 00:00:00 2001 | |
| From: Shravan Karthik <[email protected]> | |
| Date: Fri, 8 Jun 2018 15:51:05 +0530 | |
| Subject: [PATCH] phy: pcie: Set APLL reference input clock as ACSPCIE | |
| Set mux configurationt to enable ACSPCIE as APLL input clock | |
| Also set ASCPCIe in RX Mode and disable dpll_ref_m2 clock | |
| --- | |
| drivers/phy/phy-ti-pipe3.c | 16 +++++++++++++++- | |
| 1 file changed, 15 insertions(+), 1 deletion(-) | |
| diff --git a/drivers/phy/phy-ti-pipe3.c b/drivers/phy/phy-ti-pipe3.c | |
| index bf46844..f3f75f1 100644 | |
| --- a/drivers/phy/phy-ti-pipe3.c | |
| +++ b/drivers/phy/phy-ti-pipe3.c | |
| @@ -410,12 +410,14 @@ static int ti_pipe3_get_clk(struct ti_pipe3 *phy) | |
| } | |
| clk_set_rate(clk, 1500000000); | |
| +/* | |
| clk = devm_clk_get(dev, "dpll_ref_m2"); | |
| if (IS_ERR(clk)) { | |
| dev_err(dev, "unable to get dpll ref m2 clk\n"); | |
| return PTR_ERR(clk); | |
| } | |
| clk_set_rate(clk, 100000000); | |
| +*/ | |
| clk = devm_clk_get(dev, "phy-div"); | |
| if (IS_ERR(clk)) { | |
| @@ -423,7 +425,6 @@ static int ti_pipe3_get_clk(struct ti_pipe3 *phy) | |
| return PTR_ERR(clk); | |
| } | |
| clk_set_rate(clk, 100000000); | |
| - | |
| phy->div_clk = devm_clk_get(dev, "div-clk"); | |
| if (IS_ERR(phy->div_clk)) { | |
| dev_err(dev, "unable to get div-clk\n"); | |
| @@ -551,6 +552,19 @@ static int ti_pipe3_probe(struct platform_device *pdev) | |
| struct device_node *node = pdev->dev.of_node; | |
| struct device *dev = &pdev->dev; | |
| int ret; | |
| + uint32_t readval; | |
| + | |
| + void __iomem *txrxregs = ioremap(0x4a003c14, 4); | |
| + readval = (uint32_t) readl(txrxregs); | |
| + readval |= 0x20000; | |
| + writel(readval,txrxregs); | |
| + iounmap(txrxregs); | |
| + | |
| + void __iomem *muxregs = ioremap(0x4a00821c, 4); | |
| + readval = (uint32_t) readl(muxregs); | |
| + readval |= 0x180; | |
| + writel(readval,muxregs); | |
| + iounmap(muxregs); | |
| phy = devm_kzalloc(dev, sizeof(*phy), GFP_KERNEL); | |
| if (!phy) | |
| -- | |
| 2.7.4 | |
| Hi Shravan, your suggested hardware and software changes resolved our PCIe issues. The EP (TDA2XX) is now properly recognized by the Host PC as GEN2 PCIe device. We tested the EP as GEN1 and GEN2 and in both scenarios the EP device is enumerated correctly. Now that the PCIe enumeration is working we also tested DMA transfer between the EP and Host PC - and we can confirm that the DMA transfer is also working as expected. Thank you for the support. Best regards, Stefan | |