My experience is that people are rejected for no good reason apart from somebody not liking something, usually not well defined, about them.

That's what I've observed for very competent people around me as well.

Most of the companies I've worked for also rejected candidates for arbitrary reasons and also had "trouble hiring".