One interesting data point here is the Fugaku supercomputer is based around ARM's scalable vector stuff (basically Cray style variable length vectors vs short vector SIMD like AVX) and no gpu. Using HBM is a key enabler here.

I'm not sure GPUs will be displaced, looking at the difficulties Larrabe had on the driver side, but I do think we'll see more flexible alternatives becoming popular.