> We can certainly make flawed decisions based on data, but I'd argue that we're more likely to make flawed decisions with no data.
What I've seen in practice so far is that the use of telemetry has harmed software quality more than helped. It often leads developers to optimize for the wrong things and make poor design decisions. This happens because they tend to think that "the data never lies", ignoring the fact that telemetry always gives a skewed and incomplete picture.
Do you have some specific examples of this playing out?
I’ve been a product manager for products that had no telemetry, and that can be a rather undesirable place to operate, especially if you’re in the enterprise space where product changes can impact the operations of businesses.
I think it’s certainly possible to focus on the wrong things, but I don’t see that as an outcome of telemetry itself as much as an outcome of a product team that doesn’t understand the problem space or customer base.
The attributes to capture are presumably based on what teams understand to be key indicators about their app/service. I think confident incorrectness armed with bad data is just a slightly different version of a complete lack of data. Such a team was operating on whatever they imagined to be important before, and they continue to do so after, albeit with greater conviction.
But good telemetry in the hands of a good product team can be immensely beneficial for decision making and can protect customers from bad decisions. Anecdotally, my ability to pull numbers about certain attributes has been key to my ability to shut down executive pressure to make changes that would have drastically impacted customers if not for the direct evidence that it would.
I’m also not claiming that downsides don’t exist, and privacy is always my primary concern, but there are a range of outcomes based on the maturity of a team/company, and as long as the PM understands that data is not an alternative to having a relationship with customers, I think data is pretty important.
> Do you have some specific examples of this playing out?
Honestly, I can't actually remember specific examples. It's not something I dwell on. But I know that's it's happened several times that software has been made much less useful to me because features have been removed on the basis of being rarely used, ignoring the fact that even though they're rarely needed, when they are needed, they're indispensible.
More often, though, the bad telemetry-based decisions I've seen are around UI changes. Things like a laser-focus on reducing the number of clicks it takes to perform things, even though sometimes reducing the number of clicks for a thing adversely impacts the usability of it.
For bad telemetry-based UI decisions, my standout example if Firefox, although that's hardly the only one.
> But good telemetry in the hands of a good product team can be immensely beneficial for decision making and can protect customers from bad decisions.
This was actually my point of view a few years back, when telemetry started to become popular. And, as a dev who sells software commercially, I totally understand the value on that side. My experience with products that have used it, though, has shifted my view.
All that said, I do agree that it's possible to use telemetry in a way that is good for users. But I don't think it's common, and I think the reason for that is economics and human nature.
Once you start measuring a thing, that tends to become a goal rather than just a data point. And since the industry is all about maximizing velocity, that effect is even stronger. Doing proper usability studies is a slow and expensive process. Telemetry can be a useful thing as part of that process, but the tendency is to make it pretty much the entire process. That does a disservice to everybody.
> the PM understands that data is not an alternative to having a relationship with customers, I think data is pretty important.
Not just the PM. The entire company. But a relationship should be consensual, not forced. I have zero issues with opt-in telemetry. When it's not opt-in, though, it's an invasion and adversarial. I presume that's not the sort of relationship a good PM wants.
I think this is a good view and explanation. At the end of the day though, the only thing that matters is what you pointed out:
> But a relationship should be consensual, not forced. I have zero issues with opt-in telemetry. When it's not opt-in, though, it's an invasion and adversarial.
That's it. No matter how many ways you dice it - collecting data without consent or forcing opt-out is an invasion. As more and more of our lives shift to being online, our privacy and our sense of autonomy in a digital world is ever increasingly paramount.
Ideally, yes. In practice, and especially in larger shops, it’s the PM’s job to own this relationship and to make sure the important players have this understanding.
> But a relationship should be consensual, not forced.
What I've seen in practice so far is that the use of telemetry has harmed software quality more than helped. It often leads developers to optimize for the wrong things and make poor design decisions. This happens because they tend to think that "the data never lies", ignoring the fact that telemetry always gives a skewed and incomplete picture.