Why Test Automation Doesn’t Always Achieve What QA Teams Expect, and How to Move the Needle

For QA and software engineering teams, test automation is a little like new year’s resolutions: It’s easy to set lofty goals, and even to pursue them nominally. But it’s much harder to achieve meaningful results.

QA engineers might say, for example, that they want to automate a certain percentage of their tests, and they might even make nominal progress toward that goal by writing more test cases or executing automated tests more frequently.

But when you drill down into what’s actually happening, it’s often the case that the nominal gains translate to very little in the way of true value creation. More frequent automated tests don’t usually improve software delivery frequently or application quality if they are not paired with other modifications, like changes to a team’s cultural norms surrounding software testing, that increase test effectiveness and confidence.

This is a bit like what happens if you promise that, starting January 1, you’ll go to the gym more frequently and you actually do, but without scaling up the intensity of your workouts or investing in new types of training. In that case, simply increasing the number of hours you spend in the gym each week is not likely to translate to an improvement in overall health.

That’s the challenge that organizations face when it comes to test automation today. Fortunately, there are solutions, but they involve thinking a little outside the box of traditional test automation strategies.

Test automation goals vs. reality

To understand how teams can boost the value that test automation brings, let’s first unpack what organizations hope to get out of test automation, and what they actually achieve.

For most, recognizing the benefits of test automation is easy enough. They expect it to lead to faster delivery cycles, earlier detection of bugs (which in turn typically reduces the time and effort required to fix those bugs because they are usually simpler to remediate if developers catch them early) and fewer defects in deployed applications.

Hoping to achieve these goals, teams often set a target for the percentage of tests that they want to automate, with something in the range of 20-30 percent being a common goal. Then, they write the tests and begin triggering them automatically as part of their CI/CD pipelines.

In theory, this should be great. It should deliver dramatic boosts in software delivery efficiency and reliability.

The problem, though, is that merely automating tests is no guarantee of real progress. Too often, QA engineers discover that their automated tests deliver little real value due to issues like:

• Unstable test environments, which lead to inconsistent results.
• False positives or negatives that stem from poorly designed tests or errors in test coding.
• Lack of understanding of how automated tests work and what they are covering because only a handful of “automation heroes” were involved in writing the tests.
• Tests that don’t evaluate the most critical features because teams either don’t trust automation for this task, or don’t know how to automate testing of these application components.

In other words, teams experience a crisis of confidence in automated testing. They don’t trust automated tests to be reliable and so, even if the tests suggest that code is bug-free and ready for release, no one is actually confident pressing the “go” button.

This can also lead to scenarios where engineers end up testing everything manually, even if they already tested it automatically, because they won’t trust test results until they obtain them by hand.

At this point, test automation has achieved no real value at all. In fact, it has done the opposite: It has increased the burden placed on QA professionals (because they now have to write and manage automated tests alongside manual ones) and the complexity of CI/CD pipelines (which now include automated tests), but without reducing the team’s reliance on manual testing.

The root causes of test automation failure

To solve this problem, organizations need to step back and assess what’s really causing their test automation strategy to fail.

On the surface, it can be tempting to blame technical factors alone, to draw conclusions like “our code is too complex to test automatically” or “we need a better test automation framework.”

But the root causes of test automation shortcomings usually boil down to cultural and organizational challenges at least as much as technical barriers. They involve problems like:

• Limited familiarity with test automation tools and frameworks among QA engineers. This issue is especially challenging today, as new types of AI-powered test automation tools emerge that engineers have never used before.
• A reluctance to adopt new technology due to fears that it won’t work. Here again, AI-powered test automation makes this challenge even more acute, as teams may believe that AI “just can’t be trusted” with something as critical as testing.
• A heavy reliance on “heroes” to drive test automation strategies, rather than making test automation a collective responsible for the entire team.
• The expectation that test automation will result in fast, easy wins, with the consequence that when it doesn’t transform software development and QA routines overnight, teams start losing faith and fail to commit to it over the long term.

See also: RPA vs. AI Automation: Is Robotic Process Automation Being Replaced?

How to get real value from modern test automation

Given that the root causes of test automation shortcomings tend to be cultural and organizational, the solutions must also focus on changing organizational culture, specifically, the culture and expectations surrounding test automation and QA.

The following steps can help.

1) Focus on outcomes, not coverage

Instead of thinking in terms of how many tests the team automates, focus on outcome-centric metrics, like the frequency of application deployments and regression cycles. Engineers should think of test automation as a means to an end, with the “end” being improvements in software development efficiency and application quality. Increasing automation for its own sake is of no value if it doesn’t improve overall outcomes.

2) Clarify test ownership

Clearly define who “owns” automated testing. Who is responsible for writing tests, deploying them, monitoring them and interpreting results? Having clear roles is important for ensuring that automated testing doesn’t default to being the purview of just a handful of “heroes” but is instead a responsibility spread across the entire organization.

3) Make automation part of the “delivery contract”

Teams should also define formally which role test automation plays in the software “delivery contract,” meaning the set of processes that occur during development. This means establishing which types of automated tests must occur, and what the team will do in response to failed tests. Setting clear, consistent expectations in this regard helps to build confidence that automated tests are a routine, reliable part of the overall development lifecycle.

4) Shift QA from execution to architecture

Rather than thinking of the role of QA engineers as being limited to deploying and executing tests, adopt an organizational mindset that treats QA as an architectural pursuit. In other words, the purpose of QA should be defining the overall processes that optimize software quality and development efficiency. Running tests should be merely one step in that broader process.

5) Use AI to minimize toil, not replace humans

Teams are much likelier to test AI-driven automated tests when the role of the tests is to reduce toil, not remove humans from feedback loops.

Toward that end, engineers should rely on AI to automate tedious tasks like healing broken tests automatically and generating boilerplate test cases. But humans should remain in charge of which tests are executed and how to react to complex test failures.

When used in this way, AI helps to reduce noise and effort, but without undercutting confidence in automated testing because humans remain in charge.

Conclusion: Rethinking modern test automation

In short, unlocking the full power of test automation requires rethinking the role of test automation in software engineering. For too long, the focus has been on counting automated test coverage or speed. The real goal should be on analyzing the relationship between automated testing and software quality outcomes. When your teams shift their mode of thinking in this direction, test automation starts meaningfully moving the needle.