Companies seem to be increasingly preferring to use invasive telemetry and automated crash reports in lieu of actual in-house testing, and they use that same telemetry to also prioritize work. I have a strong suspicion that this is a significant contributing factor to the absurdities and general user-hostility of modern products.
I'm in complete agreement. Thanks to automated crash report uploading, the software I use is more stable than ever — it's a genuine surprise to me when an application crashes, and I can't remember the last time I had to reboot because my OS froze.
But this means that anything that's not represented in telemetry gets completely ignored. The numbers won't show you how many of your users are pissed off. They won't alert you to the majority of bugs. They won't tell you if you have a bloated web application that's stuffed full of advertising. They won't tell you if your UI is incoherent.
I really do think that large companies are looking at the numbers instead of actually using their software, and the numbers say that everything's fine.
Indeed. The way I've been summing it up recently: A/B testing is how Satan interacts with this world.
It's understandable people want to base their decisions off empirical evidence. But it's important to remember that what you measure (vs. what you don't measure) and how you measure will determine the path you're going as much as the results of these measurements.
Telemetry.
Companies seem to be increasingly preferring to use invasive telemetry and automated crash reports in lieu of actual in-house testing, and they use that same telemetry to also prioritize work. I have a strong suspicion that this is a significant contributing factor to the absurdities and general user-hostility of modern products.