Your suite isn’t slow. It’s lying to you.

Someone on your team hits re-run before they read the failure. You may have done it yourself. That single gesture, the finger reaching for the button before the eyes reach the stack trace, is the most reliable diagnostic you have. It means the suite has become an obstacle rather than an oracle. Not because anyone is careless. Because they have learned, correctly, that reading the result first is often wasted effort. That is a rational adaptation to an unreliable signal.

The tell is not a number. It is a habit. Your eventual pass rate may look healthy at the end of every run, after the second or third attempt. The coverage dashboard is green. The suite no longer has authority.

The problem is not how long the suite takes. It is that nobody trusts it. These are different problems, and fixing the first without addressing the second produces a faster way to be ignored.

What trust erosion actually costs

The obvious loss is wasted minutes. The structural loss is harder to see and considerably more expensive.

When a red build no longer stops anyone, the suite’s veto power disappears. Developers learn to route around the signal rather than respond to it. Genuine regressions begin travelling into production disguised as flaky noise, invisible because they look exactly like the results everyone has agreed to tolerate.

There is a subtler cost still. When trust erodes, it does not erode selectively. The tests that do catch bugs are discredited alongside the ones that don’t. Coverage numbers continue to look healthy while covering progressively less of what matters. The dashboard says fine. The system is dark.

Vrbo, part of the Expedia Group, documented what this looks like at scale in a public engineering write-up. Their Selenium suite’s pass rate approached roughly 50%, not meaningfully different from a coin flip, and developers were spending double the time re-running it to arrive at a build they felt confident merging. That is not a story about a slow suite. It is a story about a suite that had lost its authority, and a team doing the only rational thing available to them: running it again, hoping for better luck.

Why the damage compounds

Microsoft Research found that developers who encounter flaky tests become measurably less likely to investigate subsequent failures. The effect is not short-lived. Each false alarm raises the threshold at which a person treats a red build as worth their time.

The suite does not fail all at once. It is discredited one dismissed result at a time.

This is what makes trust erosion dangerous rather than merely inconvenient. A flaky run does not only waste the time of the run it occurs on. It degrades the response to every later failure, including the genuine ones. By the time the pattern is visible, a year of coverage numbers tells you nothing useful, and the tests that actually catch regressions are getting the same shrug as the tests that don’t.

Speed is not the problem

Duration is easy to measure and satisfying to optimise. A shorter feedback loop is valuable, but only when the signal at the end of it is trusted.

A 40-minute suite that people act on is worth considerably more than an 8-minute suite they re-run twice before believing. Applying speed to an untrusted signal does not make the signal trustworthy. It delivers the wrong answer with less delay.

Google’s internal data, widely cited in the test reliability literature, found that roughly 1.5% of test runs encounter flakiness, and around 16% of tests are affected over a sufficient time horizon. No individual team generated those figures. The accumulation is gradual and structural. Teams that discover this tend to discover it because their suite has already lost credibility, rather than as a warning ahead of the fact.

What to measure instead

These four metrics expose what duration hides. You can collect all of them from CI logs you already have, without buying anything.

• First-run pass rate, not eventual pass rate. Most dashboards report whether the suite eventually went green. The number you want is how often it goes green on the first attempt, without intervention. The gap between those two figures is your trust deficit expressed as a single number.
• Re-run frequency per build. This is the single best proxy for suite confidence. If the average build is re-run 0.1 times before merge, people trust the result. If it is re-run 0.7 times, they don’t. No survey required. The evidence is already in your logs.
• Time to first investigation of a failure. When a red build sits uninvestigated for hours, the most informative explanation is that people don’t believe it, rather than that they are too busy. Duration metrics will not surface this. A chart of median time-to-investigate will.
• Quarantine count, and age of quarantine. A quarantine list is a good-faith attempt to preserve signal by isolating noise. It becomes a graveyard when entries are never resolved. The number of quarantined tests, and how long each has sat there, tells you how much of your nominal coverage is actually dark.

Rebuilding trust in a suite is a different project from making it faster, and in most cases the more valuable one. The question is not how quickly the suite can return a result. It is whether anyone acts on the result when it does.

If you want a second pair of eyes on where your suite stands, hello@qualitylabs.eu is the right place to start.