Update 2019-06-10: This product is now available to purchase! Read the documentation here, and get it from the order page here.
Over the last few months I’ve been really busy working on a new product and I just want to take a step back today and share some of it. Let me start with what it is and then I’ll tell you about how and why I did it.
The product
A 4-port Gigabit Ethernet mezzanine card designed for Avnet’s Ultra96 Zynq Ultrascale+ single board computer.
The benefits. The low cost and small form factor of the overall system makes it possible (I hope) to not only prototype new products but to go to market with this solution until investment in a custom design is justified by market demand. The other major benefit is community support - the Ultra96 has a large and growing user base and is part of an ecosystem that is supported by the 96Boards community as well as the PYNQ community. If you want to develop a product with this hardware, you’re not going it alone and you can leverage a lot of work that has already been done.
The applications. So much is done with gigabit Ethernet these days, here are some of the applications that this product will add value to: industrial controls, network security, smart NIC, network monitoring, hardware accelerated routers/switches.
The features
The 4 Ethernet ports each connect to a separate Ethernet PHY, the DP83867IS from Texas Instruments. This is a low power, robust, high immunity gigabit Ethernet PHY with many powerful features including:
- Extra low latency (TX < 90ns, RX < 290ns)
- Wake-on-LAN packet detection
- IEEE 1588 Time stamp support
- Cable diagnostics
The mezzanine has a stacked low-speed expansion connector that brings up all of the unused low-speed I/Os of the Ultra96 and can be used to attach a second mezzanine card such as the Sensors Mezzanine. The extended I/Os include:
- 10x I/Os of the programmable logic (PL) (can be reconfigured to GPIO, UART, I2C, SPI, or other low-speed interfaces)
- 3x GPIOs of the processor system (PS)
- 2x I2C buses (I2C0 and I2C1)
- 1x SPI bus (SPI0)
- Power and reset pushbuttons
The story
I’ve been closely watching the Ultra96 since Avnet launched it early last year. Its powered by the Zynq Ultrascale+ MPSoC, it has the dimensions of a credit card and it is (at the time I write this) the lowest-price ZU+ board on the market. What’s more, it’s designed to the 96Boards spec so it can carry almost any of the standardized mezzanine boards (add-on cards) now on the market. When they then released PYNQ support later in the year, I had to get one.
Why make an Ethernet mezzanine? The marketing around the Ultra96 is mainly focused on AI, which is natural because most of the 96Boards SBCs are designed with vision applications in mind, and the ZU+ has enormous potential in accelerating neural networks. But the Ultra96 is also a great fit for Ethernet - the ZU+ has 4 internal Gigabit Ethernet MACs, and it also has the ability to run SGMII links on it’s I/O pins (SGMII is a serial PHY interface that operates at 1.25Gbps). Ethernet is also a great fit for us. Through our Ethernet FMC product, we’ve helped literally hundreds of companies develop their own Ethernet products. So an opportunity eventually fell into my lap. One of our clients was looking for exactly this kind of solution and they were willing to share the design costs. So I knew we had at least one customer, and a good part of our expenses were covered.
The design
There were quite a few design challenges to overcome to make this board happen. The constrained PCB real estate and location of the expansion connectors severely limited my choice of parts, most notably the quad RJ45. The Marvell PHYs that we use on the Ethernet FMC were not suitable for this, so I couldn’t reuse that trusted design. The pin assignment of the high-speed expansion connector on the Ultra96 wasn’t exactly ideal for Ethernet. The low I/O supply voltage meant that I couldn’t be sure that it was all going to work until I built and tested the actual hardware.
Limited PCB real estate. The obvious challenge was overcoming the physical constraints. The credit card sized form factor is great but it doesn’t provide much room for a big quad RJ45 connector and 4 Ethernet PHYs. But the limited space was not the only issue - there is almost no clearance between the bottom side of the mezzanine and the tops of the USB connectors on the Ultra96, so you can’t put components under there, and you definitely can’t have the legs of an RJ45 connector poking through. The other issue is that the MDI pins of the majority of RJ45 connectors would poke through at exactly the spot where the high speed expansion connector sits. I started by looking at surface mounted connectors, which are terrible for mechanical robustness, but I didn’t see any other way. It turned out that there were very few surface mount options available, and those that were were not suitable for various reasons. From the hundreds of thru-hole options, I ended up finding about 3 connectors that fit, but only if I squeezed two of the mounting posts between the two USB connectors. Also, because I needed the MDI pins to be well clear of the expansion connector, the RJ45 had to be especially deep, so it had to take up more PCB real estate than I would have liked to give up. Needless to say, my choice of Ethernet PHY was also strongly influenced by the limited PCB real estate - I needed the smallest package requiring the least external circuitry.
SGMII. The high-speed expansion connector of the Ultra96 has 14 differential pairs routed to it, connecting to 28 I/Os. To connect 4 Ethernet PHYs through 14 pairs, there is really only one possibility: SGMII. The Marvell PHYs that we use on Ethernet FMC have got an RGMII interface, so I couldn’t use them and I couldn’t leverage any of the Ethernet FMC’s design.
SGMII over LVDS. SGMII is a serial gigabit interface, so one interface uses only 4 pins (2 for TX and 2 for RX). Normally, running gigabit interfaces to an FPGA requires gigabit transceivers, but the pins on the expansion connector don’t route to transceivers. So I had to use “SGMII over LVDS”. SGMII over LVDS is a method for implementing SGMII on I/O pins of the FPGA, where the transceiver (data recovery, encoder/decoder) is implemented by programmable logic in the FPGA. Xilinx Vivado includes a free IP core called Ethernet PCS/PMA or SGMII that implements SGMII over LVDS.
LVDS on a 1.2V bank. The high speed expansion connector of the Ultra96 routes to pins in an I/O bank that is powered by 1.2V. Here’s the problem: you can’t use the LVDS IO standard on pins in a bank that is powered at 1.2V. The solution turned out to be to use AC coupling (required by SGMII anyway) and what they call a “pseudo differential IO standard” called DIFF_SSTL12.
Pin assignment trouble. This was the biggest risk I faced in designing this board. If you’ve ever used the Ethernet PCS/PMA or SGMII IP, on the ZU+, you would know that it requires special selection of the I/O pins - you can’t just connect it to any I/O pins. But I didn’t have that luxury, I had to use the I/O pins that were already chosen by Avnet: the pins that were routed to the high-speed expansion connector. And it turned out that they were not ideally suited for this IP core. No combination of those pins would allow me to implement 4x SGMII links. Sh#t. I told my customer that we weren’t going to get 4 ports out of this. I spent days playing around with the core in Vivado and reading the product guides. I almost retired to the idea of having only 3 ports on this board. Then I got an idea. I might have found a way to sidestep the requirement of the Ethernet PCS/PMA or SGMII IP that was preventing me from using those pins. But it was a gamble, I didn’t know if it was going to work, I couldn’t find anyone in the forums who had tried it before and I couldn’t test it until I had the hardware in my hands. I had to give it a shot. I designed the board with 4 ports, knowing that we might not be able to get all of them working. Well it turned out that I was right, I got all 4 ports working and I’m glad that I didn’t settle for 3.
Your ideas
Now I want to know what you guys think. What would you do with this compact but powerful Ethernet development platform? What mezzanines would you stack onto it? I really hope that this mezzanine delivers value to a lot of projects - what’s yours?