Simplest explanation: you start with a fully connected network of N layers. This already gives you the functionality. Then you remove a lot of unnecessary connections. This gives you the performance.