Combat just isn't the core gameplay experience though. When people talk about Zelda they talk about exploring the world, about the plot opening up, about the puzzles they got stuck on. They don't talk much about their favourite boss fights, and when they do it's usually about puzzling-out parts (figuring out what you had to do to defeat x) rather than the real-time part.

I agree that there's some difference, but we wouldn't consider every game with quick-time events to be an action game. Fundamentally it's just a small part of the experience, and not a genre-defining distinction IMO.