Azure has RDMA, though with slightly high quoted latency (I don't know the message rate), and tightly-coupled stuff appears to scale: https://techcommunity.microsoft.com/t5/azure-compute-blog/hp...

It seems how people do HPC on AWS is limited by what AWS can do (and maybe costs). Our experience was that even the elastic feature wasn't, and we often couldn't get resources anyway.

Maybe dated, but for context, we had 2TB and 128 real cores a decade ago, and I currently work with Summit-type hardware; I'd rather not admit after how long.