I think you're often correct. Literary writers seem to bring their ego to sci-fi, thinking "I'll show these plebes real writing." The result usually sounds like a Dad trying to play cool guitar licks for his teenage son's friends.

But there are counter-examples. Jonathan Lethem is at his best when he toes both worlds, and Iain "M" Banks is equally famous for his literature and his space operas. It's an absolute crime that James Morrow isn't better-known. And pretty much everything the Slate article hits is excellent.