Flaky test rate
Also called flaky tests or test flakiness
A flaky test is one that passes and fails on the same code, without anything having changed. Flaky test rate is the share of test runs — or of tests — that produce a different result on a rerun of the identical commit.
How it is measured
Two definitions, both in common use, and they answer different questions:
Per run — of all test runs in a window, what share failed and then passed on a rerun of the identical commit. This is the number that describes how much noise the team is living with day to day.
Per test — of all tests in the suite, what share have flaked at least once in a window. This is the number that describes how much of the suite is untrustworthy.
A suite can look fine on one and terrible on the other. Fifty flaky tests that each fire once a month produce a low per-run rate and a suite nobody believes.
Why it is worth watching
Flakiness is not primarily a testing problem. It is a decision-making problem, and it compounds:
It destroys the signal. The purpose of a test suite is that a red build means something. Once developers learn that red sometimes means nothing, the rational response is to rerun rather than investigate — and the first genuine regression in months gets rerun too.
It is a hidden tax on delivery. Every rerun is a delay between a change being ready and being merged, which lands directly in the review-wait and deploy-wait stages of cycle time. Teams frequently look for a delivery bottleneck in review capacity when it is actually CI reruns.
It is often a real bug. A test that fails one time in fifty because of a race is describing a race that exists. Production runs that code far more than fifty times.
What it misses
A low rate can mean a weak suite. Tests that assert almost nothing never flake. A team can drive flakiness to zero by removing every test that touches concurrency, time or integration — which is exactly the set most likely to catch something real.
It concentrates, so aggregates mislead. Flakiness is usually a power law: a few tests generate most of the failures. A suite-wide percentage makes it look like a diffuse cultural problem when it is often five tests and an afternoon.
Infrastructure gets blamed for it, and sometimes deserves to. Under-resourced CI runners produce timing failures that look identical to badly written tests. If flakiness rose the week the runners changed, the tests may be innocent.
How to read it well
Rank by test, not by suite. The list of the ten flakiest tests is directly actionable; the suite-wide rate is a temperature reading.
Watch the trend rather than the level, and watch it against suite growth. A rate holding steady while the suite doubles means flakiness is being introduced at the same rate it is being fixed.
Track quarantines as a number of their own, with an age. A quarantine list that only grows is a flaky test rate that has been hidden rather than reduced — the metric will improve and nothing will have changed.
Frequently asked questions
What is a flaky test?
A test that returns different results on the same code. Run it once and it passes; run it again with nothing changed and it fails. The result carries no information about whether the code is correct, which is what makes flakiness worse than a straightforwardly failing test — a failing test tells you something true.
What causes flaky tests?
Almost always a hidden dependency on something the test does not control: timing and race conditions, shared state left behind by another test, test ordering, real clocks and dates, network or filesystem access, randomised data with an unhandled edge case, or parallel workers competing for the same resource. Time and shared state are the two biggest sources by a wide margin.
How do you measure flakiness?
Rerun failures on the identical commit and count how many pass on the retry — those are flakes by definition. Over a window, the rate is flaky runs divided by total runs. Tracking it per test rather than in aggregate is what makes it actionable, because flakiness concentrates: a handful of tests usually produce most of the noise.
Should you delete or quarantine flaky tests?
Quarantine first, delete rarely. Moving a flaky test out of the blocking suite restores trust in CI immediately, which is the urgent problem. But a flaky test is often a real race condition that will eventually happen in production, so quarantine should come with an owner and a date, or the quarantine list becomes a graveyard of unexamined bugs.
Related
- Defect rate Defect rate is how many defects reach a given stage per unit of work — commonly bugs found in production per release, per change, or per period. The version worth tracking is the escaped defect rate: the share of defects that got past every check and reached a customer.
- Cycle time In software delivery, cycle time is how long a change takes from the moment work starts on it to the moment it reaches production. It is the internal counterpart to lead time, which starts its clock when the customer asks rather than when the team begins.
- Read: Structuring github workflows for simple reliable deployments →