And their bottleneck is what? Data transfer. State is gigantic and needs to be frequently synchronized. That's why it can only work with sophisticated, ultra high bandwidth, specialized interconnects. They employ some tricks here and there but they don't scale that well, ie. with MoE you get factor of 8 scaling and it comes at a cost of lower overall number of parameters. They of course do parallelism as much as they can at model/data/pipeline levels but it's a struggle in a setting of fastest interconnects there are on the planet. Those techniques don't transfer onto networks normal people are using, using "distrubuted" phrase to describe both is conflating those two settings with dramatically different properties. It's a bit like saying that you could make L1 or L2 cpu cache bigger by connecting multiple cpus with network cable. It doesn't work like that.
You can't scale averaging parallel runs much. You need to munch through evolutions/iterations fast.
You can't ie. start with random state, schedule parallel training averaging it all out and expect that you end up with well trained network in one step.
Every next step invalidates input state for everything and the state is gigantic.
It's dominated by huge transfers at high frequency.
You can't for example have 2x gpus connected with network cable and expect speedup. You need to put them on the same motherboard to have any gains.
SETI for example is unlike that - it can be easily distributed - partial readonly snapshot, intense computation, thin result submission.
Not disputing all of that, but telling the GP flat out "no" is incorrect, especially when distributed training and inference are the only way to run modern massive models.
Inference - you can distribute much better than training. You don't need specialized interconnects for inference.
The question was:
> > There is probably a simple answer to this question, but why isn't it possible to use a decentralized architecture like in crypto mining to train models?
> Can you copy a neural network, train each copy on a different part of the dataset, and merge them back together somehow?
The answer is flat out no.
It doesn't mean parallel computation doesn't happen. Everything, including single gpu, is massively parallel computation.
Does copying happen? Yes, but it's short lived and dominates, ie. data transfer is bottleneck and they go out of their ways to avoid it.
Distributing training in decentralized architecture fashion is not possible.
You can't scale averaging parallel runs much. You need to munch through evolutions/iterations fast.
You can't ie. start with random state, schedule parallel training averaging it all out and expect that you end up with well trained network in one step.
Every next step invalidates input state for everything and the state is gigantic.
It's dominated by huge transfers at high frequency.
You can't for example have 2x gpus connected with network cable and expect speedup. You need to put them on the same motherboard to have any gains.
SETI for example is unlike that - it can be easily distributed - partial readonly snapshot, intense computation, thin result submission.