I don’t disagree with you, but I think the human mind is essentially programmed to seek first causes. Part of what has made us so successful as a species is that we want to chase patterns down to their foundations, and I think it’s natural to feel that there is some “root” truth that explains the entirety of existence. I think that’s why a lot of people turn to simplistic unifying philosophies; it reduces a lot of intellectual anxiety if you have one big idea that subsumes all others, and upon which you can fall back when faced with uncomfortable questions. All of which is a long way of saying that while you’re correct, I think as a species we’ll always trend towards trying to find “THE foundation” even when the more rational thing is to “pick and choose” like you’re suggesting.
An interesting thought brought to mind by your comments, our search for foundations (whether set theory, type theory, category theory, or higher order) reveals as much about our own minds as it does any idea of some external nature.
I think the issue here is that if mathematics is talking about anything at all, then the search for the first cause tells us what that thing is. This is a philosophical issue separate from the sociological activity of doing mathematics. The latter doesn't need an all-encompassing foundation to continue to do useful and interesting mathematics, so pragmatics shrug the issue off. On the other hand, even if mathematics is purely about "structure", then we should still be able to organize our knowledge in a way that provides a single foundation (even if that means showing that the different formulations are logically equivalent).
One suspects that if the search for "foundations" were made rigorous enough to analyze in any meaningful way, someone like Gödel would come along with some sort of impossibility theorem. In other words, if we were to really examine the idea that any particular foundation could be the best/most natural/simplest/whatever, we'd see that it is folly.
