Beefy VRAM is a start, but Nvidia's real moat is CUDA. If PyTorch runs on AMD's ROCm, or the Intel equivalent, as well as it runs on CUDA, then we'll see some real competition here.

Chris Lattner's Mojo programming language may present an alternative here, but it's still closed source.