No, I'm not just talking about organic milk. I think there is a huge taste difference even between grass-fed organic milk and "normal" organic milk. (Note I've never seen non-organic grass-fed milk). Normal organic milk doesn't actually taste all that different compared to conventional milk to me.

For me the biggest taste difference is from the grass, not the organic.