I think there are two axes at play here - simplicity/complexity, and familiarity/alienness. Maths might be simple, but it's also incredibly alien. Things that are alien to conventional thought processes are always disliked and disregarded when possible.