Even when full source is available it doesn't really solve the problem. Many of the drivers provided for android socs are very poor quality and would not be allowed in the kernel. Typical problems include not using linux conventions for config parameters(device tree) and duplicating large portions of existing kernel functionality.

Not getting into the tree is a problem because kernel interfaces change all the time. When someone changes a kernel interface they are expected to update all of the affected code, but out-of-tree drivers don't get that.