This is one of the reasons I like matrixed organizations as it allows people, over several years, to choose who they work with and encourages leaders/managers to develop effective people and leadership skills. Siloed organizations unfortunately facilitate assholes.
Don't matrix organisations put you in the stressful position of having two or more bosses, who may disagree about what they are looking to see from you, especially when it comes to evaluating your performance and promotion/fire recommendations?
It sounds pretty good to be able to move around in an organisation, but knowing who determines your fate in the short term, and who doesn't, is really important too.
In my experience, when done well, the manager on the org-chart is primarily a people manager and does the annual performance reviews of staff based on input from the employees' project managers, peers, and/or team leads. If they have additional goals (technical/product/etc.) beyond improving the capability and impact of their group then that can definitely muddy things up for everyone.
