A bug (exploit) in the JVM is a bug, it doesn't matter if you are running it server side or client side.

The only reason it doesn't matter server-side is that you are not trying to exploit your own installation. But the bug is still there.

Of course, the JVM is the JVM wherever it runs, but when one is under the impression of a blanket statement like "Java is secure", they're likely to be thinking of server-side processes which rarely get compromised for reasons you've stated - despite having the same "level of security" wrt vulnerabilities.

You don't know that. Depending on the type of bug (for example a string overflow) simply accepting data from someone else could trigger it.