This is the local maxima problem. Any company that relies heavily on A/B testing suffers this problem. I know Netflix had/has this problem.
You try out a bunch of changes, and one of them makes a slight uptick in the metric you're tracking. So you implement that change and kill all the other experiments, instead of iterating on the other ideas to see if you can get them to do even better than the current winner.
Apple didn't used to suffer from this problem, because Steve Jobs didn't care about data if something bothered him. He'd just have it changed, and he happened to have good taste. Johnny Ive tried to do the same, but his taste was hit or miss, so now they have a conglomeration of taste and data made interfaces.
Google is 100% data made interfaces. From what I understand, their PMs have no leeway to go against the data. At least at most companies the PM can still make a decision against the data, and as long as they can justify it with user studies or something else they can move ahead.
Data can tell us a lot of things, but it can't tell us everything.
I like your description of this being a local maxima problem. The UX won't escape this maxima with incremental user testing. Users also push back on any changes to existing products' UI. Sometimes the best option is to create a new sibling product with a new UX.
> Apple didn't used to suffer from this problem, because Steve Jobs didn't care about data if something bothered him.
In his book "Creative Selection: Inside Apple's Design Process During the Golden Age of Steve Jobs", author Ken Kocienda (an early engineer on Safari and the iPhone) claims Apple doesn't do any user testing to make data-driven product design decisions. Apple designers and engineers would create many prototypes and demo them up the VP reporting chain until the best prototypes got to Steve.
Creative selection sounds like a perfect name for this phenomenon. GP's comment reminded me of working with genetic algorithms in school. Maintaining enough diversity and avoiding local maxima was a huge part of the process. Added that book to my reading list; thanks.
Yeah, combine the basic challenge of the local maxima problem with the risk-averseness of a management chain whose reputation depends on running a division with a few popular services, and you end up with a small army trying to move the other mountains to your current mountain rather than creating, fostering, killing, and growing new bets.
You try out a bunch of changes, and one of them makes a slight uptick in the metric you're tracking. So you implement that change and kill all the other experiments, instead of iterating on the other ideas to see if you can get them to do even better than the current winner.
Apple didn't used to suffer from this problem, because Steve Jobs didn't care about data if something bothered him. He'd just have it changed, and he happened to have good taste. Johnny Ive tried to do the same, but his taste was hit or miss, so now they have a conglomeration of taste and data made interfaces.
Google is 100% data made interfaces. From what I understand, their PMs have no leeway to go against the data. At least at most companies the PM can still make a decision against the data, and as long as they can justify it with user studies or something else they can move ahead.
Data can tell us a lot of things, but it can't tell us everything.