What is the cost of an equivalent setup using A100s?
I have no idea what I am doing but here goes!
By [1] we have 156 FP16 TFLOPS, taking their non "*" (* = with sparsity) value. So you need 5 So $40,000? pls the other stuff, and someone to make a profit putting it together say $50,000?
So this setup is 3 times cheaper for the same.
If I am allowed to use the sparsity value it is 1.5 times cheaper.
I like George's style and wish him well. But I'm not optimistic about their chances of selling $15k servers that are $10k in parts (or whatever the exact numbers are).
It's just too easy for anyone to throw together a Supermicro machine with 6x GPUs in it, which is what it sounds like they'll be doing.
My guess is they'll end up creating some premium extensions to the software and selling that to make money. Or maybe they can sell an enterprise cluster manager type thing that comes with support. He's good at software so it makes sense for him to sell software.
And maybe the box will sell well initially just as a "dev kit" type thing.
Have you seen what a DGXA100 costs? It starts at $199k for 8 40GB A100's, which have a list price of $10k each. So the GPU costs are $80k. What do you get for the extra $120k? 1TB ram, 2 2TB NVMe OS drives, 4 4TB NVME general storage, and 8x200Gbit infiniband. I would guess no more than 20k all of the remaining hardware. So that's a ~$100k computer selling for $200k. And that's with NVDA likely making massive margins already on the A100 and the Infiniband hardware.
The reality is that companies want to buy complete solutions, not to build and manage their own hardware. A $15k a computer that's $10k in parts is not a large markup at all for something like this.
I agree the DGXA100 is a "complete solution" because it's NVIDIA selling NVIDIA customized integrated/certified/tested/supported hardware and software.
NVIDIA's advantage is that they're a proprietary company and they're the ones actually making the chips they're putting in a box.
That's very far away from a random little open source startup slapping third-party GPUs in a generic box.
> It's just too easy for anyone to throw together a Supermicro machine with 6x GPUs in it, which is what it sounds like they'll be doing.
HPC compute is well advanced past just slapping GPUs into generic supermicro servers anyway. Without semi-custom hardware and equivalents to nvlink/nvswitch AMD won't ever be competitive in the HPC space.
We can't really comment much on it because a bunch of specs are lacking. Is it using 6x 7900 XTXs? Which Epyc CPU (Epycs vary in price from $1K to $11K)?
The tinybox
738 FP16 TFLOPS
144 GB GPU RAM
5.76 TB/s RAM bandwidth
30 GB/s model load bandwidth (big llama loads in around 4 seconds)
AMD EPYC CPU
1600W (one 120V outlet)
Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks)
$15,000