The automation project that found eight bugs in three years

The rarest document in test automation is not a success story. It is a named quality director, writing under her own name, publishing a detailed account of a three-year programme that failed. Carina Gerry, Director of Engineering Quality at SalesLoft, did exactly that in May 2020 with a post called “Automation: A Failure Story.” The title does not hedge.

It is worth crediting her for this. Post-mortems from individuals willing to name the project, the timeline and the yield are uncommon in this field. Most automation write-ups describe success. This one describes a suite that ran for three years, cost significant maintenance effort, and found eight bugs. That takes something to publish.

Seventeen testers and a swarm

By 2015, SalesLoft had two QA engineers handling releases. By autumn 2016 the team had grown to five. Eventually it reached seventeen manual exploratory QA engineers. Regression was handled through what Gerry describes as “swarms”: concentrated manual efforts lasting one to two days before major releases.

The swarm model scaled well enough to keep growing alongside the product, but it made every major release a significant team event. Automation appeared to be the obvious answer, and at the time it was broadly the consensus answer across the industry.

The suite they called LARS

The team built a browser regression suite called LARS, using Capybara, Selenium and Ruby, running on AWS EC2. Over time it grew to roughly 300 tests.

Three hundred tests is not a trivial undertaking. Writing them takes months. Maintaining them requires sustained attention to environments, dependencies and infrastructure. By the time LARS reached that size, it was carrying a meaningful share of the QA team’s time.

Two problems persisted throughout the suite’s life. The first was runtime: the full suite took over an hour to complete. The second was reliability: the tests were flaky, mostly because they were written against brittle CSS selectors. When the UI changed, selectors broke. When selectors broke, the suite needed triaging before anyone could trust its results.

An hour-long suite changes how it fits into the development cycle. You cannot run it before committing a change, or before opening a pull request. You run it after work is already merged, which means failures arrive as interruptions to something you have already moved on from. It stops being a feedback mechanism and becomes a delayed alarm.

Three hundred tests, eight bugs

LARS found three to four bugs in its early period, and eight bugs in total across roughly three years of operation.

That number is the one Carina Gerry reports, and it is worth sitting with. Three hundred tests, over an hour per run, maintained for three years, and the total yield was eight bugs. On those numbers, the cost of the suite almost certainly exceeded the value it delivered. Yet the team kept investing in it rather than cutting it. The reason the author gives is direct: no success metrics were ever defined for the project. Without a target, there was no basis for comparison and no agreed trigger for reassessment. The project could not be steered towards success because nobody had agreed on what success looked like, and it could not be cut early because there was no threshold it had visibly failed to meet.

Then the architecture moved. SalesLoft migrated from Angular to React and split into microservices. The UI team shifted from Ruby to JavaScript. LARS was left behind in both senses: the product it tested had become substantially different, and the engineers writing the new code could not maintain a Ruby test suite. Ownership had quietly transferred to nobody.

The suite died on 23 January 2019. The team returned to the seventeen manual exploratory QA engineers.

“Catch bugs daily prior to them hitting production.”

That is how Gerry describes what the manual team does. It is not a consolation; it is a result.

A structural failure, not a technical one

Looking at the decisions made along the way, each was defensible at the time. Capybara and Selenium were the standard tools for browser automation in Ruby shops in 2015. CSS selectors were how most people located elements in that period. Running on EC2 was a reasonable infrastructure choice. Building browser tests for a web application is not, in itself, a mistake.

What went wrong was not any single technical choice. It was the combination: no definition of success; tests written in a language the UI team did not own; selectors coupled to markup that changed with every redesign; and a runtime that placed the suite outside the natural development rhythm. Each problem was individually tolerable. Together, they ensured the suite would drift further from the code under test with every passing month, with no agreed threshold for deciding it was no longer worth the investment.

Three years is also long enough for a product to become a substantially different thing. Any automation programme measured in years is racing the architecture it was written against, and the architecture moves continuously. The suite was already fighting obsolescence well before the Angular-to-React migration made it final.

Was going back to manual wrong?

The comfortable reading of this story is that the automation failed because of implementation errors, and that a better implementation would have succeeded. That reading is worth challenging.

Seventeen exploratory testers catching bugs daily is not a consolation prize. Exploratory testing with skilled practitioners finds the kinds of failures that scripted tests miss: unexpected interactions, context-dependent behaviour, things that look wrong to a person but pass a binary assertion. LARS found eight bugs in three years. The manual team found bugs daily. On those numbers, the manual practice was the more effective one.

The lesson here is not that the team should have automated harder, built better selectors, or rewritten the suite in JavaScript when the UI team switched languages. It is that automating browser interactions across a moving architecture, with no definition of success and no clear ownership, does not reliably beat a competent manual team for a product that is changing quickly. Automation is not always the right answer. On these facts, returning to the model that was genuinely working was a rational decision.

This conclusion cuts against the commercial interest of any firm that sells automation consultancy, including this one. We are stating it anyway because the evidence points to it.