The 3 A’s: Agile, Automation, and AI

A strange, uneasy feeling came over me while I chatted with a project manager about an upcoming ecommerce project. As we talked about the project’s goals, expectations from the team, the timeline, and the challenges the project team will face, the stranger I started to feel. Looking at the chicken scratches on my iPad, which passed as my notes, the source of the unease jumped out in three bullet points: Agile, Automation, and AI.

1. The Business expected the team to use Agile to reduce the project’s timeline, shifting from the waterfall approach used with prior projects.
2. Automation would be used to run more functional tests in less time with a stretch goal of using fewer manual testers once the automation was up and running.
3. Test data, which takes more time to prepare than running the test, would be handed off to an Artificial Intelligence (AI) to generate for us.

These three elements can make or break your project. After an hour of discussion, I got the question I was kind of dreading, “So what do you think?” “There’s a problem here…” I started, fully knowing my answer wouldn’t be the bow of agreement the project manager was hoping for. “All of these things will actually add time to the project if we’re not careful. Because if you look at Agile, Automation, and AI closely, you’ll see that while each will add time at the beginning of your project, there are opportunities to make the time back up if the team makes the right decisions.” With a slight sigh, the project manager said simply, “So where are the risks, and where do we make the time back up?”

It’s been over 20 years since the Agile Manifesto was first posted and we are still arguing about what it really means in practice. At its core, Agile saves time since you are delivering business value in intervals over the course of the project rather than waiting for the whole thing to be delivered at the end. This is where you’re able to save time by moving to Agile – by delivering business value sooner.  But having practiced agile since 2006 using four different methodologies (Dynamic Systems Development Methodology, Scrum, Kanban, and Scaled Agile Framework 2.0), I can tell you the two things that nobody accounts for when shifting to any agile-based framework.

The first risk is to assume that just because we say we’re agile as a team doesn’t mean the rest of the organization understands the shift that’s needed to support the team. I was on an Agile team where our product owner, a key stakeholder for any Agile team as they determine if the team’s work has delivered business value as expected, didn’t want to be part of the daily standup meeting because they were too busy supporting their three other projects. However, the product owner would often express frustration as to why they were ‘out of the loop’ when delays happened, and would schedule ‘catch-up’ meetings that set the team further behind. Ensuring everyone supporting the team is aware of the commitment and has the time needed for the project is critical for Agile’s time savings to ever kick in.

The second thing to consider is the time it takes for the team to shift to a business-value mindset from a feature mindset. Teams can get caught in the trap of thinking that shifting to Agile means that business value = feature.  An example would be if the team built an API and thought they had provided value to the business. While you’ll need the API as part of the solution, the value to the business has yet to be delivered. In this case, the API would need to be connected to a user interface that the business could use to interact with it. Business value only comes when the team delivers something that the business can use to address a need.

Conversations around Automation – specifically functional test automation aka where the automation tool is mimicking what the user is doing – start and stop with the question, “What do we want the automation to tell us when it’s complete?” Do we want to see if the user got back the right results based on their actions? Are all the graphical elements on the page where we expect them to be? Did the page load as fast as we expected it? Or a collection of all of the above? Yet, each of these tests is different from the others and, in some cases, would need a completely different tool to find the answer. This also assumes that your user interface is complete because any UI change can and will break your test automation script causing you to rewrite it. It’s a lot, but we can whittle it down by going back to the original question; what will we know when the automation tests are complete?

One of the best functional test automation suites I got to work with answered the question this way; we want to make sure that customers can order our #1 item. Sounds simple, but to accomplish this they had to do two things that were time-consuming at first.

1. The first was building a script for each step the customer had to do to place the order. So one script checked the product detail page for the item. Another would add the item to the cart ensuring the targeted size and color were selected. Three more scripts handled the entry of the customer’s shipping address plus payment information. And the last script checked the order confirmation screen. By the time these five scripts ran, the team knew if any of their changes broke the most valuable flow for the business. Also by having five individual scripts instead of one large script, it allowed them to quickly isolate any issues the scripts found as well as make any updates to accommodate new UI changes.
2. The second thing was to build a set of ‘gold test data’ of the inventory records that would return to their starting condition when the test automation completed its run and the data changes were validated. While the use of gold test data isn’t new – it’s been used for years in database and reporting testing – its importance when setting up any test automation can’t be understated. I’ve learned the hard way that trying to ‘use what we have’ from a data standpoint is a great way to burn too much time during what is already a very time-consuming phase of the project. You also have the added bonus of missing bugs on the side.

AI for me can be summed up by my favorite William Gibson quote, “The future is already here – it’s just not evenly distributed.” Part of the reason for that is some of the first uses of AI on an enterprise level have been with automation tool vendors such as Eggplant and Tricentis to help build/maintain scripts and Broadcom. CA’s Agile Requirements Designer, which can help you map your test cases to your test data, tells you how any changes will ripple through both. While Large Language Models (LLM) are all the rage for what they can do – giving you the side eye Chat GPT – you still need to be sure that you have a clear set of goals for what you want these tools to do in addition to an understanding that you might get some off results. After all, AI is being trained on internet data; the good, bad, and ugly of it all.

One way that I’ve used Chat GPT to save time in my testing is to, oddly enough, generate faux customer data that I can use for placing orders. If you are looking for tips on how to implement AI, there are some great lists in the QA community such as Beth Marshall’s Beth the Tester Tales and Ajay Balamurugadas at The Test Tribe to help you get started. Just note that like with automation, you will need to put in a lot of time up front to save time later.

As I was summing all of this up, the Project Manager sighed on the other end of the call. Then they said, “You know, I just have enough time to do the minimum.” But before they could go on I asked, “Did I tell you about the Aries MACH Booster?” 

As we have gotten deeper into the MACH framework with our clients, we’ve leveraged many of these ideas about agile and automation so we can help deliver your next ecommerce solution within 100 days of kickoff. In many respects, Aries Solutions’ experience with commercetools, combined with our years of experience seeing what works and doesn’t work from a project leadership and testing perspective, helped build the MACH Booster. So why not let us boost your next project with it?