On the contrary, with AWS et al, it's just too easy to have noisy neighbors :)

I don't think many people are intentionally deploying their apps with replicas of different sizes, you're right.

But this is it: noisy neighbours, inherent physical differences in even identical hardware, using different node sizes in your clusters. I think incidental differences in servers are very common, even within the same AWS instance category.

I thought of another variant: with AWS autoscaling groups using spot instances you can list a number of different instance sizes and say "give me whatever is cheapest" and you'll often get a mix.