Correct. If your game can't cope with Linux "fragmentation" (most of which is already abstracted away by Steam, so the remaining "fragmentation" is with hardware, which is the same problem you have with Windows), then you're in for a world of hurt if you try to port to a console with its far-from-ordinary hardware and programming APIs and such.
Porting to consoles is much easier than Linux, as each console is a very fixed platform. In my (limited) experience, middleware like Unity works better on consoles than it does on Linux.
