It might help to take a look at Arista/Metamako 10G L1 switches for some inspiration. These are 48x ports and use the FPGA for specific network applications (including switching). You might be able to find some of the older models on the cheap on eBay.
I’ve opened up a few and the board itself seems fairly simplistic. I do recall a giant copper heatsink though.
The board-level architecture is going to be super simple as big FPGA designs go:
* XCKU5P in the middle
* 12x GTYs routed to 4x Samtec ARF6 connector for the line cards
* 2x GTYs routed to 2x SFP28 uplinks
* RGMII to back panel management PHY
* Parallel SRAM bus to STM32H735 management processor
* A bunch of Murata MYMGK modules for power conversion off the 12V rail
* STM32L431 in QFN48 or more likely BGA100 depending on IO requirements as a PMIC and to manage reset sequencing etc
This will be fully FPGA based, no separate switch ASIC, and I want to do all of the hardware design. I'm not sure there is much I can learn from somebody else's FPGA switch design at the board level - it's basically just going to be a bunch of transceivers hooked up to SFPs and some power distribution. All the magic happens inside the FPGA.
I’ve opened up a few and the board itself seems fairly simplistic. I do recall a giant copper heatsink though.