Zalando load tests in production. Here is why.

From 4,200 to 7,200 in one year

Zalando is one of Europe’s largest fashion platforms, and Cyber Week is the moment that defines its engineering year. In October 2020, executive principal engineer Bartosz Ocytko published a detailed account of how Zalando prepared for the season on the Zalando Engineering Blog. The headline figure from 2019: measured peak orders per minute reached 7,200, up from 4,200 in 2018. That is a 71% year-on-year increase in the sharpest moment of load. Annual customer growth was running at 20 to 25%, with 840,000 new customers that year alone.

At that trajectory, the capacity planning question changes. It is no longer whether your checkout service can handle the traffic. The question is whether it can still handle it when every other service it calls is simultaneously under peak load, with interaction effects that nobody has modelled because they have never occurred at the same time before.

The limit of staging

The standard answer is a staging environment: spin up a copy of production, drive synthetic traffic at it, observe what breaks. The problem is that a staging environment is a model of your production system, not a copy of it. At Zalando’s scale, the interactions between a thousand services under simultaneous peak load are not reproducible outside the real platform. Dependencies, latency profiles, buffer sizes, shared queues: none of these are faithfully replicated in staging, and at sufficient scale the divergence becomes the whole story.

Ocytko’s team reached a conclusion that is quoted more often than any other line in the load testing field:

“The only way we found effective for a large-scale system like ours are live load tests in production.”

This is not a preference or a convenience. It is a constraint of the architecture, stated plainly after years of running both approaches and comparing the results.

What production testing actually requires

Deciding to test in production is the easy part. The programme Zalando runs has three non-negotiable requirements, and each is harder than it sounds.

The first is synthetic orders that can be separated from real ones at the fulfilment stage. Zalando places test-product orders against the live production system and separates them from genuine customer orders further down the pipeline, before anything real is dispatched. That tag must survive every handoff. The testing is real; the fulfilment is not. This modification to the fulfilment pipeline is infrastructure work that has to be done before the test can run at all.

The second is coordination across roughly 100 on-call teams who all know the test is happening. Most capacity planning programmes do not budget for this. If an engineer’s service alerts during a live load test and nobody told them to expect it, the programme is not testing the platform. It is generating false incidents and spending the goodwill that will be needed when real ones arrive.

The third is the number that deserves the most attention. Zalando tested 1,122 applications in their programme, out of more than 4,000 on the platform. Most capacity planning tests the three services that everyone already suspects might break. The interesting failure modes live in the services that nobody thought to check, because nothing about their normal behaviour suggested they were a risk. Testing 1,122 applications is not fastidiousness for its own sake. It is the only way to find the service that has never been on anyone’s risk register and falls over under load it has never seen before.

The tool choice: maintainability over raw performance

In a second post published in March 2021, Zalando engineers Amila Kumaranayaka and Carsten Timpert described how they moved from manually triggered load tests to a fully automated system running on Kubernetes. Their tool evaluation covered JMeter, Vegeta and Locust. They chose Locust.

The reason is worth quoting directly: Locust was “more popular within our development teams, thus the test suite would be easier to maintain.” Not the most performant option. Not the most feature-complete. The one their teams could actually work with, understand, and keep current over time.

This is an underrated principle in every part of testing. A tool that is technically superior but that nobody on the team can edit without two hours of context recovery is not superior in practice. The earlier approach, manually triggered tests, was described by the team as “cumbersome and expensive.” Automation brought it into the normal engineering workflow. For payment load tests specifically, they used Hoverfly to mock external payment dependencies, with each test run taking roughly two hours. The choice of Locust was what made that maintainable after the initial build.

Upstream dependencies: Spotify’s 2022 Wrapped

Zalando’s lesson is about what happens inside a large platform. Spotify’s experience with 2022 Wrapped, described in a March 2023 post on the Spotify Engineering Blog, adds a different dimension: what happens when the bottleneck is not a service you own.

Wrapped is one of Spotify’s highest-traffic annual events. Their 2022 edition had more than 150 million engaged users across 111 markets, with tens of millions arriving in the first three to four hours after launch. The team expected tens of thousands of requests per second simultaneously across US, EU and Asia regions.

Spotify’s own services were load-tested. But Wrapped sits on top of many services that the Wrapped team does not operate. The preparation work was to identify every upstream service dependency, coordinate with each of those teams, confirm that each had provisioned for the anticipated load, and verify that the provisioning was complete before launch day. The launch had no significant technical issues.

A service that passes its own load test while the upstream services it depends on have not been provisioned for the event has not actually been tested end to end. Mapping the dependency tree before a load event, including the teams you do not control, is not optional once you reach a certain level of integration.

What Ticketmaster has not published

Ticketmaster has not published an engineering postmortem for either of its two most-cited incidents. What follows is based solely on Ticketmaster’s own public statements, US Senate testimony, and third-party reporting, and should be read in that light.

For the Taylor Swift Eras Tour presale in November 2022, Ticketmaster’s own public statements and testimony to the US Senate put the scale at 3.5 billion system requests in a single day, four times their previous recorded peak, with 14 million simultaneous users and bots on the platform. company-reported figures cite 2.4 million tickets sold; the general sale was subsequently cancelled. The failure mode visible to users was a queue counter that moved backwards.

Cockroach Labs published a blog post analysing the incident. That analysis is vendor commentary from a distributed-database company with a direct commercial interest in the conclusions it reaches. It may be accurate; it cannot be independently verified, and it should be treated as such.

For the Oasis ticket sale in August 2024, The Stack, a technology publication, reported widespread 503 and 500 errors and an anti-bot system that flagged legitimate buyers as suspected automated traffic. No technical postmortem followed that reporting.

The more durable observation is not technical. Nobody outside Ticketmaster knows the actual root cause of either failure, because Ticketmaster has published nothing that would allow the field to learn from it. The queue that ran backwards, the error codes, the flagged legitimate users: these are visible symptoms. The engineering cause remains undocumented. A company at that scale, after failures of that visibility, choosing not to publish a postmortem is itself a data point. Resist the impulse to diagnose the root cause from the outside. What we can say is that in both cases the system did not gracefully reject new requests once capacity was exceeded, and that every technical explanation in circulation comes from parties who were not in the room.