Proposal to tweak DQS pin-mapping on future Au V2+

Hi folks,

I’m tinkering with building a custom DDR memory-controller from scratch on the Au V2. I’m making progress, but as I try to take a more-robust approach, I’ve found some issues.

One issue stems from the fact that the SDRAM DQS lines are not bound to clock-capable pins (SRCC/MRCC). This causes trouble for routing to BUFIO, which is needed to drive ISERDES/CLK. I’ve found that I can route through BUFMRs to BUFR/BUFIO, but there are only 2 per clock-region, and I also need BUFR/BUFIO for the ISERDES/OSERDES output-clock and clockdiv.

I was able to resolve these problems, but it still might be nice to have the DQS on CC pins. It looks like this could be arranged by just swapping the 4 DQS pins (2x p/n) with other nearby pins. For MRCCs, we could swap with D13/C13, E12/E13 (currently mapped to: ddr_reset/ddr_dq4, ddr_a10/ddr_ba0). These replacement pins are right next the the BUFIO/BUFR arrays in the X0Y1 clock-region, where all the other DDR I/O is also placed. For SRCCs, we could swap with C11/C12, E11/D11 (currently ddr_dq3/ddr_dq6, ddr_we/ddr_ras), which are also in the same clock-region as the other DDR I/O pins.

I think the appropriate spec is in fig 3-63 on page 105 of “7 Series FPGAs Packaging and Pinout Product Specification” (UG475) AMD Technical Information Portal I’m comparing that with the Au V2 schematic, and the device-view in Vivado.

I’m pretty new to this stuff, so maybe I’m missing some obvious reason for having the pin assignments the way they are.

Anyhow, ISERDES/OSERDES turn out to have other problems for me, so I’ve made things easier for myself by moving back to IDDR/ODDR, which is much simpler. But it might still be nice to be going through CC pins, for better control of jitter and more-compact routing.

The tools have very specific requirements for the DDR pinout to be able to use the built in controller. If you look at the IO pins, some are labeled as DQS which are the clock capable pins to the memory controller. They don’t overlap with xRCC pins.

• DQS signals for a byte group must be connected to a designated DQS pair in the bank
due to the dedicated strobe connections for DDR2 and DDR3 SDRAM. For more
information, see 7 Series FPGAs Clocking Resources User Guide (UG472) [Ref 10].

See page 193 https://www.xilinx.com/support/documents/ip_documentation/mig_7series/v4_1/ug586_7Series_MIS.pdf

Thanks for responding. Sorry for the long delay.

I saw that there are pins labeled for DQS connection on the chip. These pins are not “clock-capable” in a sense recognized by the router.

get_property IS_CLK_CAPABLE [get_package_pins B15]
0

What I’m trying to do is probably not very common. Most people would just use the MIG to get an interface to SDRAM. But if you want to build one from scratch, you have a problem because the DQS pins are not Clock Capable, so they can’t be routed directly to BUFIOs. For DDR reads through IDDR/ISERDES, this means each DQS read-clock (UDQS/LDQS) has to be routed by some other means to the 8 corresponding ILOGIC cells, next to the individual IOBs.

If they can’t be connected directly to a BUFIO, then they can’t be routed with low latency and jitter directly to the IDDRs for the data DQ lines. Here are some solutions.

(1) Just do what the MIG does

The MIG uses the hardware PHASER_{IN/OUT}_PHY units. Each DQS drives a PHASER_IN_PHY, which then drives ISERDES clocks on the associated DQ lines.

https://docs.amd.com/r/en-US/ug586_7Series_MIS/Overall-PHY-Architecture

As far as I can tell, these units are inaccessible to us outsiders. There are no primitives. So, this seems unworkable.

(2) Route the DQS through BUFR → BUFIO → IDDR/C(B). This has the advantage that each DQS clock arrives at roughly the same time at all the IDDRs. Unfortunately, it also means there’s a latency of >5 ns in the clock signal. For any reasonable memory clock-rate, this far exceeds the delay that can possibly be added to the DDR data coming to the IDDR/D from the IOBUFDS through the local IDELAY. So, this seems unworkable.

[Idea: Maybe I could just depend on there always being 1 “wasted” clock of SDRAM read-preamble (no data), and I could just plan to always lose the first DQS clock-cycle of (empty) read-data, and the IDDRs would simply latch the actual DQ data (after preamble) with the long-delayed DQS read-clock. That requires tuning the DQS->IDDR latency to be exactly 1 clock cycle, which seems like it would boil down to requirement for a single possible SDRAM clock-frequency.]

(3) Route the DQS clocks directly to the IDDR/C(B) pins. This works, but it has significant variation in latency. I couldn’t find constraints that made any difference in how these lines are routed. (I tried set_min/max_delay and set_bus_skew, but all were ignored. Maybe I have some other constraints that are interfering?) The latency on these DQS clock routes is large enough that it seems to require adding a DELAY on the individual DQ lines, but also varied enough that (for reasonably-fast clocks) it also requires a different custom delay on each line.

These custom delays can be computed from examining the timing-paths on the DQS lines to IDDRs. Then the processing of parallel input (bytes) from SDRAM reads downstream of the IDDRs also requires custom delays to decompensate the delays upstream of the IDDRs, but that seems like it should be workable with constraints, after measuring the (different) timing-paths from DQS to the downstream parallel work.

(4) Maybe there are simpler/smarter approaches? I’m new to this stuff, and I’m open to learning from others. Maybe my goal is just not reasonable?

(5) Revised Suggestion:

Your comment about the need to connect SDRAM to the specified DQS pins on the chip makes sense, and that’s probably what the vast majority of your customers want. I don’t know whether it is difficult to do (e.g., expensive, complicated, technically difficult, too much time needed), but would it be feasible to add a simple solder-able jumper on the board, so that users could select between normal SDRAM pin-connections and DQS->CC connections? It’s a great board, anyhow.

Thanks.

I actually just tried to wire up the UberDDR3 open source driver last night, and it worked pretty easily right out of the box. I did it because I wanted to do some latency measurements and experiments where I took manual control of refresh, which the Xilinx MIG doesn’t let you do. And it worked! I was able to get basically 12-cycle (at 100MHz) worst-case read latency for random access fetches, presuming that refresh was handled manually during non-critical windows, and done often enough to keep up with the required refresh rate. Which for my particular use case is quite easy to achieve, and being able to guarantee low-latency reads during my actual critical periods is a design requirement.

But the point is that all worked completely fine even though it’s just using the stock DDR3 pin assignments that the Au uses, and the driver didn’t have any clock-capable pin issues. So I guess, the real question is “why are you trying a custom DDR driver from scratch?” If it’s for your own education/entertainment, that’s awesome! But if it’s for some other reason like “the standard MIG sucks, particularly for random access latency”, then maybe you should have a look at UberDDR3.

But I should note that the UberDDR3 doesn’t “let you” take manual control of refresh, but because it is entirely open source, it’s possible to relatively easily patch it in order to take that control.

Even if your goal is to roll your own, it might still be very educational to have a look at what UberDDR3 is doing, particularly if you get stuck otherwise. A working reference implementation is a great learning aid.

2 Likes

Many thanks for the comments. I have started looking at UberDDR3. That might be the best way for me. Your comments are useful.

My goals are some mix of: eliminating dependency on IP and proprietary (opaque/untouchable) details, supporting optimizations for specialized data-flows, reducing complexity/resources/power, and of course learning. Ultimately, I’m exploring whether I could use a collection of Au boards for a prototype of a multiprocessor design. Still so much to learn and wrestle with.

For various reasons, I’m hoping to run with an SDR clock in the neighborhood of 350 MHz. I know that’s fast for a hand-built project, but it seems this target is feasible, based on an almost-working approach I had a while back. If I was more-sophisticated, I’d be doing proper read/write calibrations to tune the delays. That would ease the demands on getting everything tuned statically, which is somewhat vulnerable to the squishy PAR process, though the simplicity of my implementation limits those effects.

I’m about ready to punt this phase of challenge, and maybe the UberDDR3 will be the relief route. Thanks.

Well an open reference implementation like UberDDR3 will definitely help on the IP angle! I am far from an expert on this, I’m doing it for my own education as well. But my read so far is that something like 350MHz is the kind of thing that you’re only going to get away with for specific, carefully designed subtasks where you can pipeline the absolute fuck out of your critical path. Anything where the nature of the problem is not innately extremely pipelined is going to be hard-to-impossible to do at that clock rate, on this kind of hobbyist-level FPGA.

Something that might get in your way is that you explicitly don’t have access to the timing primitives you need for doing full read/write calibration on the DDR3 bank. The primitive that the MIG uses is explicitly opaque and undocumented. Although who knows, maybe you can talk an LLM into reverse engineering it, which would be completely awesome.

350MHz feels extremely challenging to target for anything but very specific, tightly pipelined workflows, on the 28nm process that we have available at this hobbyist level. Me, I’m new to this too, and I was definitely sweating it in my last design where I was finding 133MHz already to be more restrictive than I wanted.

1 Like

Well, it’s ugly and finicky, but I am trying a static calibration of a highly-pipelined READ via a manual process involving scripts that access timing-paths and generate per-DQ delay-taps and constraints. There’s also the impl timing-sim, the LEDs, and occasionally the ILA.

1 Like