SQA

Qaptly is here!

·6 min read

Finally, it's time!

v0.13.0 is out today

iOS, which had stalled for two whole major versions, finally gets stability support like never before, and can at least see Android's tail lights now.

Half of it was that iOS support had to catch up. The other half was that we were forced into it. v.0.12.15 was released to go with the splash window in our second open-source tool, Qaptly Auto for Android, and after auto-updating, almost every standby machine online stopped working. We had released the new core-layer Qaptly CLI (with splash window support) and Qaptly Auto for Android in the wrong order, so the old one got packaged. So, just as the iOS onboarding pages were finished, we did basic functional verification and released v0.13.0, so that every machine in the standby pool now supports remote iOS testing.

v0.13.0 released

How Serene (Qaptly's predecessor) began

Back in 2024, we already had two AI products running, and Serene was still an unstructured prototype. But it was my personal favourite. My first proper job in life was related to it, and although I left that role many years ago, I still think back to it now and then and miss the thrill of finding a problem.

In August, we finally decided to put the team's main effort behind this idea. At first, we focused on turning product specs (in all their forms) into test plans and test cases, so the whole system was built around analysing and breaking down information, reassembling it with SQA-specific knowledge, and generating test cases from that. Before this we had only done small-scale tests: on simple flows, our fine-tuned engine reached over 80% accuracy and coverage (compared against human-verified test cases). At that point there was only a backend, and a very simple one. To this day, that repo is still called serene_tmp_backend.

From September to November, a major client went through an initial validation of the product model with us. At that stage our system was basically data moving back and forth between various storage layers, with no interface to be seen.

The target was a highly flexible membership system serving more than 400 brand clients. At acceptance, as I recall, the pass rate of the test cases was under 70%, so the client eventually dropped this direction. Still, allow me a little hindsight: I manually worked through every one of those test cases at least three times, and across the scenarios of nearly 200 cases our error rate was actually under 2%. I even rewrote a copy of their membership system just so these cases would not embarrass us when we ran them. I was quite surprised by the acceptance result. All I can say is that PMs really need to write good specs, otherwise the specs and the actual system will not match.

That client really was great, though. They said it was fine that this did not work out, and asked us to help with automation instead!

I thought, well, it is on our roadmap anyway, so let's do it!

From roughly December to March this year, we worked almost entirely on automation. Vision LLMs had just come out, and as a way to verify tests they were frankly a disaster. Besides Vision LLMs, we also built many CV and pattern matching algorithms, and did a great deal of workarounds and extra engineering to fill the gap, until we finally got through the problems the client had given us.

Before acceptance, we repeatedly stressed to the acceptance testers that they must make sure the verification environment was stable, and not run it on a personal computer.

The test results did not surprise us: they were run on personal work computers. I would not say that is wrong, but to make these end-to-end tests run reliably, we had built quite a few assumptions into the system design, and one of the most important was a stable environment. All that CV locked automatically onto the resolution of the machine it first ran on. Looking back at the results later, the results from three machines with different resolutions did not match.

What can I say? We still did not do it well enough. But that is also why, for a long time, browsers were where our test support was most complete.

At the time, the product very nearly got shut down. After all, we had failed to win over the major client we had in mind, twice!

Fortunately, some friends were later willing to try our services, and by funding the product with services, we slowly made it through.

How much faster can it be?

Fast forward to Lunar New Year 2025. An old friend I had once started a company with (he had gone back to being a CIO at a Fortune 500 company) was chatting with me on the phone...

When a tech enthusiast becomes a manager, the questions can sting.

"How much faster than a person can you make this? If it's not two or three times faster, don't bother, right?"

Over the past two years, every time I hear someone say that AI + Playwright is all you need for testing, I want to ask: did you write your test cases yourselves or did AI write them? Is your model one you fine-tuned yourselves, or one someone else trained?

Why ask that?

Most senior SQA engineers will understand. Today's models are like a very clever stranger: they talk convincingly, but they can only do very junior SQA work.

In test execution, the two things that matter most are, first, being able to reproduce the scenario, and second, confirming whether the scenario matches the expected result. Being even slightly off is not acceptable.

Take a case we handled recently. We asked a VLLM whether the word "收件" ("receive") appeared on the screen. One character too many or too few is wrong.

We tried every mainstream LLM API, and every one of them said yes. But the screen actually only showed "去收件" ("go receive"). So if the design shows "收件", and every related spec and acceptance criterion says "收件", is "去收件" right or wrong?

I bring all this up because I knew about this bottleneck back then: without targeted fine-tuning, it could not replace the SQA role.

This is how I answered my friend: "Right now it's probably less than 30%, but I'll make it 10 times faster, and half the cost!"

I am still some way from that goal, but you have to have a dream, right? What if it comes true? Dream Big, Achieve More

Looking back

After shipping v0.13.0 today, I can breathe a little easier and start on the next step.

As of today, we are serving or have served 13 clients across 17 product lines, and these products have more than ten million users in production.

Qaptly has 8 separate repos in total, covering backend, frontend, client, Android, iOS, macOS and more,

with 176 releases and 3269 commits in total. On our busiest day, across backend, frontend and client, we knocked out 31 Jira tickets with more than 100 commits!

Finally...

- Clients who cannot write any code can use Qaptly themselves to automate mobile / Web / API testing

- Write test cases and automation just by talking and moving your mouse

- GUI automation used to copy the form; now it can grasp the intent

- AI handles fallback and failover for you when test cases run

- Use the client's own GPU to speed up testing; even 16G is enough to go into battle

- We open-sourced a few small tools we think are pretty useful

We have probably poured more than 20 years of software development experience into this project. Of course, we think there is still plenty we could do better, but it is also time for this tool to be used by more people.

L.L.

Need Professional Assistance?

Our team are senior engineers and former executives with 20+ years in commercial product development — from apps serving hundreds of millions of users to B2B SaaS platforms serving tens of thousands of merchants. In recent years the focus has been enterprise AI adoption, from mapping the opportunity through to tuning what is live. Whether that means:

  • Mapping AI opportunities and sequencing them
  • Preparing data flows and integrating existing systems
  • Building forecasting and optimisation models
  • Monitoring and tuning after go-live
  • Test strategy and process review
  • Building and adopting test automation

We can provide professional advice and concrete solutions for teams of different scales and types. If you're facing similar challenges, feel free tocontact our consulting team.