What a real Selenium migration timeline looks like

The situation they were in

In September 2024, Zenjob’s engineering team published an account of their Selenium-to-Playwright migration on the Zenjob Tech Blog. The numbers they led with were the appealing ones: a suite that had taken 35 minutes to run now runs in roughly 7 minutes.

The suite was not large. One hundred test cases, distributed across eight parallel Jenkins jobs. By most organisations' standards, that is a small-to-medium suite. Plenty of vendors would quote a fortnight to migrate it.

What drove the migration was not speed, at least not primarily. The team’s post is clear: flakiness in the Selenium suite “persisted” despite added synchronisation and dedicated test data, and required “constant rework”. Their setup also lacked built-in reporting. The engineers were not just running tests; they were managing a support process around those tests to keep them useful at all.

What they actually did

Before writing a single Playwright test, Zenjob wrote an RFC. They compared Selenium, Cypress and Playwright, set out their reasoning, and circulated the document to the whole engineering team for comment. Only after that process concluded did they begin the migration.

This is worth dwelling on. An RFC is not bureaucracy. It is the mechanism by which a migration earns its budget and builds its mandate. Every engineer who will later review Playwright tests, maintain them, or question them during an incident has already been asked to engage with the reasoning. That is a meaningful thing. The social cost of a migration (the time and energy required to align a whole organisation around a new approach) is real, and it almost never appears in anyone’s timeline estimate.

Once the RFC was settled, the team ran both suites in parallel: existing Selenium tests kept running while Playwright tests were written, with the balance shifting slowly until the old suite could be retired. The post describes them “slowly increasing Playwright tests and reducing Selenium tests”. This is the careful approach. It avoids a single cutover and means the pipeline is never entirely dependent on a suite the team has not yet fully validated.

It is also expensive, because for months you are maintaining two suites simultaneously.

What it cost them

Eight months. For one hundred tests.

That figure deserves to be the headline of this piece rather than a footnote. Zenjob did not take eight months because they were disorganised or slow. They took eight months because this is what a migration actually costs when done carefully: writing and circulating the RFC, running both suites in parallel, keeping the pipeline green while the product continued to ship, handling the surprises that turn up mid-migration, and eventually retiring the Selenium suite gracefully.

Vendors selling automated conversion tools tend to quote timelines measured in days. Some framework comparisons published online use even shorter windows for suites of similar size. It is worth asking, calmly, what those timelines include. They often cover the mechanical translation of test syntax. They rarely cover the parallel-running period, the test data rework, or the weeks that follow when engineers encounter a new framework for the first time on a live pipeline and edge cases surface one at a time.

The chart below shows the execution time before and after, as reported by the team.

After the migration, the pipeline ran eight parallel workers; local runs used two. The team reports an 80 percent reduction in test execution time, from 35 minutes to roughly 7 minutes. measured

There was one notable problem during the migration. Playwright’s fullyParallel configuration option, which activates full parallelism across all test files, caused some tests to fail. The team disabled it. Once disabled, the post reports, the tests achieved “stable performance”.

This is a useful concrete lesson. fullyParallel is a headline feature of Playwright, and frequently cited when making the case for the migration. It is also a feature built and tested against greenfield suites that have been designed for parallel execution from the start. A migrated suite carries assumptions from its previous life: about execution order, about shared state, about test data that was set up with sequential runs in mind. Turning on fullyParallel against a migrated suite is, in practice, an experiment. Zenjob ran the experiment and found it did not work. Their fix was to turn the feature off, which is a reasonable outcome. It is worth noting, though, that the post’s speed figures reflect a suite running without full parallelism enabled.

The number they did not publish

The stated driver of the migration was flakiness. The introduction describes flakiness persisting despite synchronisation improvements and dedicated test data, and characterises it as requiring constant rework. It is clearly a significant part of the reason the team invested in a migration at all.

The post does not publish a post-migration flakiness rate.

This is not a criticism of the team or of the write-up. Flakiness is genuinely difficult to measure in a way that produces a comparable number. Speed is straightforward: you compare timestamps before and after. Flakiness requires you to define what counts as a flake, choose a measurement window long enough to be representative, and control for changes in the suite itself during the measurement period. Speed gives you a clean before-and-after; flakiness rarely does.

But the gap is worth naming as a pattern. Teams migrate partly because of flakiness, and report on speed because speed is measurable and the number is good. If you are evaluating a migration, the question “how much faster is the suite” has a clear answer. The question “how much calmer is the maintenance burden” almost never does, even though it is usually the more important one.