Team Perspectives

Is using AI the line between development teams that survive and those that don't?

·4 min read

Three years ago we started using AI tools in our development process, and nobody was sure what they would change. Looking back now, the real dividing line may not be who uses which tool, but who is willing to change how they work.

Then and now

This morning, in a meeting, I happened to walk a partner through how we have changed over the last two years. It left an impression on me afterwards, so I am sharing it here.

From 960 hours to 120 hours

This compares two requirements of roughly the same size. In mid-2023, an experienced 8-person development team that had worked with us for a long time put in two straight weeks under close to 996 pressure (the 9am-to-9pm, six-days-a-week schedule), for (8x12x10) hours of work. Last month, the same kind of job took about 120 hours (4x5x6), and, importantly, with fewer bugs.

Coding went from 73% of the time to 3%

Most of those 120 hours actually went into polishing screen flows and adjusting the specification. All of the business logic was written by AI.

What did we do?

Gave every developer access to AI tools

This sounds ordinary. Surely every boss does this already!

Here is how our team is set up. Everyone has at least ChatGPT Pro + Claude Max + GitHub Copilot Pro, and until July this year everyone also had Cursor. On top of that, our Google Workspace tier includes a fair amount of Gemini usage, so we use that too. Not counting PMs and designers, each person's AI development accounts cost at least US$150 a month.

Made developers and processes adapt to the AI tools

Shouldn't you pick AI tools that suit your people? Why make the people adapt instead?

We learned this from a partnership that was about to be signed. After we lost that deal, our internal post-mortem found that the problem was not that our product fell short. The other side's R&D staff, during acceptance, had deliberately picked the system's limitations at the time as grounds to object. Strictly speaking, our product already exceeded the output and quality their own people were producing. Once we had checked and confirmed that conclusion several times, we reached an internal consensus: use it first, and objections don't count.

Lots of internal resistance? Of course. Anything that threatens people's jobs meets resistance. We set our position early: if you can't stand it, that's fine, you're free to go. Fortunately everyone was already used to working this way, adapted very quickly, and nobody left.

New hires must be able to develop with AI tools

What does "be able to" mean? Saying you can use ChatGPT in an interview doesn't count!

We give you a test account and ask you to set it up yourself, then within two hours build an MVP from vague requirements, something that used to take 1-2 weeks. It has to run to pass! Why stress that the requirements are vague? Because with clear requirements there is no room for thinking. The candidate could paste them into any tool, and you could not tell whether they really know how to use it.

Can't finish it? Then don't say you can use these tools. They have had this capability for more than a year. If you still can't use them, don't claim to be a fast learner. However capable you are, if you don't learn, it's no use!

A fine-tuned whisper

We have a lot of internal terminology. Without fine-tuning, no model really speeds up the process, because you still have to correct the transcript afterwards. So we run our own whisper model, deployed in the cloud, with an internal app dedicated to the job. It is not as convenient as Realtime Chat, but for turning our own requirements into text it is more than enough.

What else do we plan to do?

A unified solution for document management

Documents are now produced so fast that our old way of managing them no longer works. Besides the shared serena we use for project context management, deciding which documents go into the context for a given round of development needs something that can tie them together. We are building this system internally, and if it works well we will probably open-source it later.

Effort estimation

Our whole estimation framework used to be built around people. After AI arrived, we knew things would get a lot faster, but how much faster was hard to estimate. We are now pulling together the boundaries of what AI can do and how time is distributed. If things go well, we may have a more accurate way to divide and estimate work by November.

Redefining roles

Because AI speeds up everyone's growth, in both depth and breadth, people have started thinking across disciplines and taking on more responsibility in the workflow. So the way we collaborate has to change too. Fortunately, change is now normal, and everyone enjoys the feeling of getting better every day.

Soundproof space

Now that even our R&D engineers write code by talking, good audio pickup and not disturbing each other matter more. Beyond better headsets, a traditional workspace doesn't really suit this, and we are still looking at different options. In the meantime, everyone who came back to the office from WFH has gone back to WFH.

Not using AI is fine too. It just costs more

You only feel the pain when you see the comparison. Once efficiency has grown 8x, and you see there is still plenty of room to improve, and people no longer have to grind through overtime, you can't go back to a slow development pace. Time is money, after all, and people cost far more than AI.

Need Professional Assistance?

Our team are senior engineers and former executives with 20+ years in commercial product development — from apps serving hundreds of millions of users to B2B SaaS platforms serving tens of thousands of merchants. In recent years the focus has been enterprise AI adoption, from mapping the opportunity through to tuning what is live. Whether that means:

  • Mapping AI opportunities and sequencing them
  • Preparing data flows and integrating existing systems
  • Building forecasting and optimisation models
  • Monitoring and tuning after go-live
  • Test strategy and process review
  • Building and adopting test automation

We can provide professional advice and concrete solutions for teams of different scales and types. If you're facing similar challenges, feel free tocontact our consulting team.