I Think the intuition is that queuing time is asymmetric around the median, so when you increase variance you also increase the mean.

Put another way: if the median is 1 minute, then half the time you'll wait less than 1 minute. The other half you might wait not just for 2 minutes, but significantly more. They won't "cancel out" to preserve the mean.