Editor’s note (2026): I wrote this in 2020, when Bamboo and Selenium were my daily tools. The tools changed; the idea that test automation lives in an ecosystem didn’t.
Originally published on Medium on 14 April 2020.
Sustainability. A trending term used in various contexts. That’s not surprising, since we are having trouble maintaining our own planet. But what is sustainability in the context of test automation? Why is it so important and why is it so hard to achieve it?
Through my experiences at different clients, my perspective on test automation changed. I have learned to see the bigger picture, and that the context determines the best approach on test automation. This is not a story with a silver bullet, but one that, I hope, inspires you to look from a different point of view and get a step closer to sustainable test automation.
What is sustainable test automation?
Is the application working as expected? Getting fast feedback on that question is the added value of test automation. Is the answer no? Then we can quickly act to keep the costs of a bug as low as possible. That is why a lot of companies invest in test automation.
Sustainable test automation is the ability to maintain test automation. More the art of maintaining test automation if you’d ask me. Setting up test automation can be a challenge, but maintaining it? A whole different story. I hear a lot of stories about that and can share some myself.
When I started at a client in 2017, my team said: ‘Anaïs, it’s nice you are our new test automation engineer, but we don’t need you. We already have a test set of 3500 tests, which is running daily!’. And I thought: an easy job, awesome! Unfortunately, I cheered too soon.
It appeared that 500 tests were failing during every daily run. It was too much to fix, so the team didn’t look at the test results anymore. Even incidents that had been identified by the test set, were going through to production. That was the point that the test set didn’t add any value. I wondered how much the team invested in this test set…
The garden of test automation
Visualizing the test results via a build orchestrator (Bamboo), helped me convincing the team to do something about this test set. I called it the garden of test automation, and the motto was ‘Keep it green!’. Together we rolled up our sleeves, we had some gardening to do.
A garden of test automation in Bamboo. Keep it green!
I pulled out my garden toolkit, filled with strategies, methods and tips which I gathered over the years. I advocated about Risk based testing and the Test automation pyramid. Based on risk we split up the tests to structurally improve them with several good practices as guidelines. Wherever possible, we pushed GUI tests down towards unit and integration test layers, or we shoveled the tests out when they didn’t add enough value.
The result was 600 tests, of which 1 or 2 tests were failing sporadically. Overall a result to be proud of, but the fight of maintaining test automation wasn’t over. It remained a big challenge. Not only for me but I’ve seen it everywhere around me, it has been a struggle for years. Why is this gardening so hard?
October 2018 I started my first assignment as a test automation coach. My goal was to help ten teams within a complex government organization getting test automation to the next level. Soon I realized pulling strategies and tricks from my garden toolkit wouldn’t be enough, I needed a different approach.
It was overwhelming, and I actually didn’t know where to start. I did know that I had to get to know the context and teams better in order to help them. Drinking coffee with lots of people was a start, but not enough to get to know the teams. I decided to go on a ‘Walk in the park’ to visit the teams.
‘Walk in the park’
The goal I wanted to achieve with this ‘Walk in the park’ was two-fold:
- Alignment on the vision and strategy of test automation
- Acquire the context and status of test automation within the team…
… Not just the tester(s), but the whole team. Because, in order to achieve built-in-quality, everyone in the team is responsible for quality. That was part of the vision I shared and discussed with the team. Was this vision and strategy applicable for this team? This question helped to create awareness and input for the retrospective.
Quality is everyone’s responsibility. (Often attributed to W. Edwards Deming.)
A retrospective is a way to centralize the team and retrospect on their work. I stole the idea from the so-called agile event, and asked the questions regarding test automation; what goes well, what are the improvement points and obstacles you come across?
From garden to ecosystem
This ‘Walk in the park’ gave quite some input on the context and status of test automation at the different teams. I mapped this input and found out that there were a lot of aspects surrounding and impacting the gardens of test automation. Turns out, the world is bigger than the gardens of test automation, it is an ecosystem of test automation!
Ecosystem of test automation
It starts with the organization climate where goals and budgets for IT originate. Often these goals involve agile working and automating the software delivery process, impacting the way of working, tooling and culture. Test automation must fit in this picture.
In addition, the surrounding environment greatly affects the test automation gardens. The teams I helped work in different technology stacks and application landscapes (Java, .NET and diverse package solutions), resulting in different tool stacks and architecture to take into account. And what about the gardeners of test automation? What if they have zero experience with programming and test automation?
Through experience we gardeners fill our garden toolkit with useful strategies and good practices. But is every tool right and applicable to any context? Would I introduce a tool like Selenium Webdriver or TestCafé to a group of testers who are very reluctant to code? And how do you apply the Test Automation Pyramid in a Siebel (package solution) environment? All of these questions… hmm… I would expect this ‘Walk in the park’ to be easy!
The world is bigger than the garden of test automation. It is a whole ecosystem with lots of aspects that impact test automation. And like in a real ecosystem, all of these aspects must be balanced to maintain sustainability. Look at the climate and environment around you or take a ‘Walk in the park’ to expand on your perspective. Anticipate on that perspective to get a step closer to sustainable test automation.
🌱 Want to read more about the Ecosystem of Test Automation? Check out my next blog; dive deep into the environment, and let me take you on an adventure through a diversity of application landscapes, shapeshifting test automation strategies and the tools used!