Never simply hope that things will go smoothly. Read everything positively, and think in reverse. (Master Sheng Yen)
The 25th of every odd-numbered month is draw day for Taiwan's uniform invoice lottery, where every purchase receipt doubles as a lottery ticket. Many receipt-related systems and apps offer ways to check your receipts, and whether they can quickly match everything and tell users their results once the winning numbers and winner lists are published looks simple. Doing it well actually involves a lot of detail, and it says a lot about each vendor's technical ability. Today, after several rounds of adjustment, one system has just been certified as able to handle several hundred thousand winning users, so this seems a good moment to talk about a scenario most SQA people rarely run into.
I opened with Master Sheng Yen's line mainly because I came across it by chance, and the first sentence sounded a lot like the development team on this project. A very optimistic team that kept thinking positively after falling over several times, which is admirable. With professional help, they will certainly go further.
A self-inflicted DDoS
This example is about checking receipt lottery results. Let us look at how the problem happened.
The product manager saw that after every draw, users had to open the app and tap to check their receipts before they knew whether they had won. So he asked the development team to let users know straight away when a receipt had won: after the draw, send a push notification to winning users, and when a user taps it, take them to the results screen in the app, where they can see the result without tapping to check again. It was a very good product insight.
Users' receipts and winning information were already stored in the database, and the app had always worked normally for them, so the back-end engineers did not think twice. Once the winning information had been calculated in the back end, they pushed notifications to all users at once. Then most of the users who got the notification, just as the product manager had imagined, logged in to see their results within the same short hour or two. And after that, nothing more happened...
For the rest of that day, users could not connect to the service. As we said above, the engineers had designed a mechanism that sent every user to the server to check their results within a fairly short time, and with no preparation, the first wave of users took the servers down.
How we finally solved this is part of another of our professional services, software architecture design, and I will leave that for a separate article. This one is about how, from a QA perspective, you should discover and verify whether a system's capacity meets expectations.
Stress testing and volume testing
First, a note on terminology. I could not find a good Chinese translation for Volume Test. The original meaning is about large amounts rather than capacity. The Chinese word we use for it can mean capacity too, but semantically the test is about the ability to handle a large volume of requests. I really could not find a suitable term, so in Chinese I still follow some of my seniors in translating it as "capacity testing".
Because they are not familiar with the definitions, many beginners come to believe these two kinds of testing are really the same thing. So why does professional software testing keep the two terms separate? And why, in so many settings today, do we only ever hear about stress testing?
Let us go back to how most SQA courses define them.
Stress testing specifically refers to verifying a system or program's ability to operate normally and handle errors when resources are insufficient.
Volume testing specifically refers to verifying whether a system or program can operate normally or handle errors when processing a "large volume" of requests. The large volume here may be a single heavy workload, such as a large file or a request that creates a large amount of work, or it may be many small requests flooding in within a short time.
It is not hard to see why people habitually mix up these two kinds of testing. Under a large volume of client requests, resources run out, so it counts as a stress test. Indeed, most development teams understand it this way. But if it were that simple, why define them separately? To make this easier to explain, this article focuses on the scenario of many requests at once. Single requests with a heavy workload we will cover another time.
Put simply, the two tests look from different angles. Stress testing actually requires SQA to understand every detail of how the system works before they can design the tests. For example, most SQA people will not know the scaling policy driven by the load balancer, or how production logs are handled and what the rotation policies are, let alone conditions like the cache mechanism and cache slot size (or count) that can push a system to its limits. Better still, most developers are not familiar with these either, so there is no matching fault tolerance in the source code. That is normal. Even senior engineers rarely run into these situations. In a normal development process nobody raises them, so you simply do not know they exist. Even when they do happen, most engineers deal with them crudely: if it is full, clear it or buy more space; if resources run short, add machines. Only a few teams with real technical discipline insist on getting these details right. This is also why many systems are fine through acceptance and before launch, and then fall over as soon as they go live.
I have wandered off again... Back to volume testing. This looks from the outside. Based on users' core flows, it simulates a large number of users (or requests) entering the system at the same time and checks whether it can stay "presentable". As long as the service keeps running, reliably serves users within the designed capacity, and makes other users wait or try again later, that is basically a pass. If, in that situation, it can scale out quickly and take on users beyond the designed capacity, so much better. But that is a real test of DevOps or SRE skill, and usually it takes a few incidents and some advice from experienced people before you know how to handle it.
Common mistakes in stress testing
Back to the topic of this article. This approach, using a large number of requests in a short time to test whether a system works normally, has one critical design condition: it only works if you pass through every system interaction in the "core flow". In plain terms, you must be able to fully simulate every step a user goes through. That sounds simple, but most teams' so-called "stress tests" do not actually test the situation they meant to test, precisely because they overlook "every step". Below are three common mistakes, along with ways you can check for them yourself.
Mistaking a part for the whole
The mistake: Reducing stress testing to sending a large number of requests, while actually only hitting some of the API calls or simply browsing the home page, without covering the full user experience (interactive flows such as logging in, searching, transacting and logging out).
The result: The bottlenecks found are skewed and the core flow is never tested, so it is easy to miss the real breaking point.
The check: Extract the decision points in the core flow, and confirm the branches that most users will go down.
A leaf over the eyes
The mistake: Many teams run their stress test scripts from several instances, or even a single instance, all in the same network segment and with the same cloud provider. In reality this only measures the outbound performance of nodes in that environment or that network segment.
The result: The total volume tested looks large, but when the same volume arrives for real, the system still goes down.
The check: Define the unit and duration for concurrent requests, set up matching monitoring, and verify against the volume per unit of time that was actually received.
Not seeing the real mountain
The mistake: Running stress tests on UAT/DEV or on an architecture different from production, or estimating load by scaling proportionally.
The result: Everything that should be done was done, but production still cannot take the load.
The check: Do not save on short-term verification costs (you really can shut it down as soon as the test is done) by assuming these similar, or even completely different, environments will perform the same. Make the environment as close to identical or similar as you can, and think about how to save on testing costs from that starting point.
Got it. What next?
That was a bit too technical, so here are some more practical ways to manage it.
If you are the client, check that the stress testing approach in the vendor's test plan is designed around the expected usage for each scenario, and that the acceptance report includes statistics on the load each path node in the scenario handled during testing. As long as the final results meet your usage expectations, it basically passes.
If you are a product development team, following the customer journey map to trace stress test paths is the most direct approach. If the team does not have one, tracing behaviour paths in GA4 (or another behaviour tracking platform), or the back-end API requests distribution, is also worth trying. What matters is that the technical team understands the order of this data and how representative it is.
Still not sure what to do after reading this?
Recently I have had some emails and messages saying that after reading my articles people still do not know what to do, and asking whether I could run a course or come to their company and assess things directly. In fact, most teams in Taiwan are unlikely to face what this article describes. Mostly the client has the final say, and if it passes acceptance, it passes. If you are getting ready to develop your own product, or you have raised it internally several times and still cannot solve it, that is a good time to come to us.