Saucelabs are kings of the "100% outage for 5% of our users = 95% availability" status update.
In particular I'd see repeatable problems where they couldn't launch whatever browser X operating system in under 2 minutes (when our tests would time out) and list allocation time as 'elevated' (say, 8s average vs their normal of 3s). If you start believing your own statistics you get into almost as much trouble as believing your own PR.
For an 18 month period where they were particularly bad, I think they only copped to an actual problem one time out of around a dozen cases where our CD pipeline was blocked for half a day or more unless we just turned off e2e tests entirely.
Per user guarantee levels. i.e 100% outage for 1 user is equivalent to a 0% availability guarantee.
The relationship between client and service is the same irrespective of the number of clients the service has - a number which means exactly nothing to the client.
But more importantly, availability numbers are for informing a client about how much incidents outside their house can affect them, and reasonable courses of action to take when it does.
When using the numbers internally, the fudged number is equally misleading. Unfortunately there exist fewer adversarial relationships internal to an organisation to prevent these short sighted statistical nonsenses.
In particular I'd see repeatable problems where they couldn't launch whatever browser X operating system in under 2 minutes (when our tests would time out) and list allocation time as 'elevated' (say, 8s average vs their normal of 3s). If you start believing your own statistics you get into almost as much trouble as believing your own PR.
For an 18 month period where they were particularly bad, I think they only copped to an actual problem one time out of around a dozen cases where our CD pipeline was blocked for half a day or more unless we just turned off e2e tests entirely.