Proposal to tweak DQS pin-mapping on future Au V2+

Hi folks,

I’m tinkering with building a custom DDR memory-controller from scratch on the Au V2. I’m making progress, but as I try to take a more-robust approach, I’ve found some issues.

One issue stems from the fact that the SDRAM DQS lines are not bound to clock-capable pins (SRCC/MRCC). This causes trouble for routing to BUFIO, which is needed to drive ISERDES/CLK. I’ve found that I can route through BUFMRs to BUFR/BUFIO, but there are only 2 per clock-region, and I also need BUFR/BUFIO for the ISERDES/OSERDES output-clock and clockdiv.

I was able to resolve these problems, but it still might be nice to have the DQS on CC pins. It looks like this could be arranged by just swapping the 4 DQS pins (2x p/n) with other nearby pins. For MRCCs, we could swap with D13/C13, E12/E13 (currently mapped to: ddr_reset/ddr_dq4, ddr_a10/ddr_ba0). These replacement pins are right next the the BUFIO/BUFR arrays in the X0Y1 clock-region, where all the other DDR I/O is also placed. For SRCCs, we could swap with C11/C12, E11/D11 (currently ddr_dq3/ddr_dq6, ddr_we/ddr_ras), which are also in the same clock-region as the other DDR I/O pins.

I think the appropriate spec is in fig 3-63 on page 105 of “7 Series FPGAs Packaging and Pinout Product Specification” (UG475) AMD Technical Information Portal I’m comparing that with the Au V2 schematic, and the device-view in Vivado.

I’m pretty new to this stuff, so maybe I’m missing some obvious reason for having the pin assignments the way they are.

Anyhow, ISERDES/OSERDES turn out to have other problems for me, so I’ve made things easier for myself by moving back to IDDR/ODDR, which is much simpler. But it might still be nice to be going through CC pins, for better control of jitter and more-compact routing.

The tools have very specific requirements for the DDR pinout to be able to use the built in controller. If you look at the IO pins, some are labeled as DQS which are the clock capable pins to the memory controller. They don’t overlap with xRCC pins.

• DQS signals for a byte group must be connected to a designated DQS pair in the bank
due to the dedicated strobe connections for DDR2 and DDR3 SDRAM. For more
information, see 7 Series FPGAs Clocking Resources User Guide (UG472) [Ref 10].

See page 193 https://www.xilinx.com/support/documents/ip_documentation/mig_7series/v4_1/ug586_7Series_MIS.pdf

Thanks for responding. Sorry for the long delay.

I saw that there are pins labeled for DQS connection on the chip. These pins are not “clock-capable” in a sense recognized by the router.

get_property IS_CLK_CAPABLE [get_package_pins B15]
0

What I’m trying to do is probably not very common. Most people would just use the MIG to get an interface to SDRAM. But if you want to build one from scratch, you have a problem because the DQS pins are not Clock Capable, so they can’t be routed directly to BUFIOs. For DDR reads through IDDR/ISERDES, this means each DQS read-clock (UDQS/LDQS) has to be routed by some other means to the 8 corresponding ILOGIC cells, next to the individual IOBs.

If they can’t be connected directly to a BUFIO, then they can’t be routed with low latency and jitter directly to the IDDRs for the data DQ lines. Here are some solutions.

(1) Just do what the MIG does

The MIG uses the hardware PHASER_{IN/OUT}_PHY units. Each DQS drives a PHASER_IN_PHY, which then drives ISERDES clocks on the associated DQ lines.

https://docs.amd.com/r/en-US/ug586_7Series_MIS/Overall-PHY-Architecture

As far as I can tell, these units are inaccessible to us outsiders. There are no primitives. So, this seems unworkable.

(2) Route the DQS through BUFR → BUFIO → IDDR/C(B). This has the advantage that each DQS clock arrives at roughly the same time at all the IDDRs. Unfortunately, it also means there’s a latency of >5 ns in the clock signal. For any reasonable memory clock-rate, this far exceeds the delay that can possibly be added to the DDR data coming to the IDDR/D from the IOBUFDS through the local IDELAY. So, this seems unworkable.

[Idea: Maybe I could just depend on there always being 1 “wasted” clock of SDRAM read-preamble (no data), and I could just plan to always lose the first DQS clock-cycle of (empty) read-data, and the IDDRs would simply latch the actual DQ data (after preamble) with the long-delayed DQS read-clock. That requires tuning the DQS->IDDR latency to be exactly 1 clock cycle, which seems like it would boil down to requirement for a single possible SDRAM clock-frequency.]

(3) Route the DQS clocks directly to the IDDR/C(B) pins. This works, but it has significant variation in latency. I couldn’t find constraints that made any difference in how these lines are routed. (I tried set_min/max_delay and set_bus_skew, but all were ignored. Maybe I have some other constraints that are interfering?) The latency on these DQS clock routes is large enough that it seems to require adding a DELAY on the individual DQ lines, but also varied enough that (for reasonably-fast clocks) it also requires a different custom delay on each line.

These custom delays can be computed from examining the timing-paths on the DQS lines to IDDRs. Then the processing of parallel input (bytes) from SDRAM reads downstream of the IDDRs also requires custom delays to decompensate the delays upstream of the IDDRs, but that seems like it should be workable with constraints, after measuring the (different) timing-paths from DQS to the downstream parallel work.

(4) Maybe there are simpler/smarter approaches? I’m new to this stuff, and I’m open to learning from others. Maybe my goal is just not reasonable?

(5) Revised Suggestion:

Your comment about the need to connect SDRAM to the specified DQS pins on the chip makes sense, and that’s probably what the vast majority of your customers want. I don’t know whether it is difficult to do (e.g., expensive, complicated, technically difficult, too much time needed), but would it be feasible to add a simple solder-able jumper on the board, so that users could select between normal SDRAM pin-connections and DQS->CC connections? It’s a great board, anyhow.

Thanks.

I actually just tried to wire up the UberDDR3 open source driver last night, and it worked pretty easily right out of the box. I did it because I wanted to do some latency measurements and experiments where I took manual control of refresh, which the Xilinx MIG doesn’t let you do. And it worked! I was able to get basically 12-cycle (at 100MHz) worst-case read latency for random access fetches, presuming that refresh was handled manually during non-critical windows, and done often enough to keep up with the required refresh rate. Which for my particular use case is quite easy to achieve, and being able to guarantee low-latency reads during my actual critical periods is a design requirement.

But the point is that all worked completely fine even though it’s just using the stock DDR3 pin assignments that the Au uses, and the driver didn’t have any clock-capable pin issues. So I guess, the real question is “why are you trying a custom DDR driver from scratch?” If it’s for your own education/entertainment, that’s awesome! But if it’s for some other reason like “the standard MIG sucks, particularly for random access latency”, then maybe you should have a look at UberDDR3.

But I should note that the UberDDR3 doesn’t “let you” take manual control of refresh, but because it is entirely open source, it’s possible to relatively easily patch it in order to take that control.

Even if your goal is to roll your own, it might still be very educational to have a look at what UberDDR3 is doing, particularly if you get stuck otherwise. A working reference implementation is a great learning aid.