WHY AUTOMATION TESTS ARE FAILING
We all know that test automation can work wonders for your business. It can help to reduce cost, save time and effort, get quicker feedback and thereby directly contributing to increased productivity. But in order to get good results from automation testing, you need to avoid mistakes because they can result in wastage of automation efforts.
To make your test automation strategy successful, you have to avoid some common mistakes & failures. With that in mind, here’re some of the common problems with automation testing that you need to avoid to build a long-term successful test automation strategy.
Going for 100% test coverage with automation.
It’s necessary to understand the importance that not everything can can be automated. One of the main benefits of test automation is to free up time for testing that must be done manually, i.e. regression or exploratory testing. In other words, it’s about applying testers’ skills to where they provide the most value.
Automation testing is not there to replace manual testing but rather to help manual testers to free up their time. Sometimes, automating those tests brings minimal returns, or human intervention is required to execute those tests.
As far as test automation is concerned, there are plenty of ways to go about setting up your test automation suite. You could use a test frame work for testing for cross browser testing & testing functionalities involving any browser interaction. If your team is not skilled with test frameworks like TestCafe, Cypress or Selenium, you can even opt for Play & Record testing to achieve some automation level.
Test automation takes time, and going big from the beginning is not recommended. It would help if you asked the following questions to yourself and your team before moving to test automation:
What pain points encountered in manual testing does test automation solve?
What kind of tests should be automated (Regression, Sanity, Functional, etc.)?
What is the test automation team’s skill-set, and is there any training required before jumping on the test automation bandwagon?
Which type of tests should be part of the CI/CD pipeline?
Just automating all manual test cases is one of the major reasons for automation testing failures since that task is ideally not achievable!
Plan for gradually automating your test suite. I recommend starting out by focusing on the flows that are easiest to automate. In most cases, you will find that it’s the relatively simple and very repetitive flows that, by far, take up most of your testing time.
Testing for multiple things in one test case.
A test case that is built to verify more than one aspect can fail in multiple ways, for that very reason. This means that if the test case fails, it must be manually inspected to figure out which aspect failed. Sometimes businesses wants to check multiple assertions or validations in one test in order to expand coverage or save time and effort, but the longer a test is running, it increases the chances for timeouts.
Build test cases so that they logically only test one aspect. This way, there is no doubt about what goes wrong when a test case fails. Instead of bundling up multiple tests in one test case, it is best practice to build reusable components with your test. This way, it is easy to reuse the logic contained in other test cases, and the time required to create a new test case is minimised.
Using a tool that is not a good technical fit
When looking for a test automation platform, make sure it fits your technical requirements.
A common mistake is to choose a tool based only on your current challenges and automation objectives. This comes with the risk of being stuck with a tool that is not a good technical fit for your business in the long run. I have experienced where management want to use automation frameworks because it’s popular or because other big corporate companies are using them. This a where the advice of an experienced automation engineer is very valuable.
Solution: Implementing test automation is a long term strategic choice and should be handled as such. When evaluating automation tools, look across your business and identify all the applications and technologies that could be potential targets for automation. Identify the scenarios where test cases need to move between technologies, e.g. both web and desktop applications, and select an automation platform that has matching capabilities. And don’t forget cross browsing compatibility.
Inadequate Infrastructure
This is one of the major factors for automation testing failure. Whether you are developing web products for the end-consumers, it would involve interaction with the web browsers. During the early stages of product testing, your team might be using a manual testing approach, which has many limitations. You cannot attain good test coverage since the number of tests may be limited or may not challenge the test infrastructure.
Take the case of cross browser testing; your team can verify your product against limited combinations of web browsers, operating systems, and devices. You can test on browsers installed on your system or machines connected to the network, but you can still not cover the entire spectrum of web browsers (and their different versions). The scenario will change when you shift the focus to automation testing or automated cross browser testing.
When you shift to automation testing, the number of tests being executed will increase significantly. To get the most out of automation, the test infrastructure on which the tests are executed should be responsive but having a responsive test infrastructure comes at a cost. The cost will rise as the product development & test phase progresses; not investing in the infrastructure will hamper the throughput of testing, leading to automation testing failure.
Your management team should take a conscious decision on whether to have a local test infrastructure or shift to a cloud-based testing where you can leverage the advantages of parallel testing for improved test efficiency.
Test Automation In Silos
It is a wrong notion that testing is a separate task. Testing and development should be done hand-in-hand. It should be a part of the development cycle and continuous integration process, i.e., Continuous Integration/Continuous Delivery pipeline. One of the major reasons for an automation testing failure is that the test teams work in silos and have little or no interaction with the other teams in the project. For more on Continuous Integration, read my blog post on the Importance of Continuous Integration For Testing.
Wouldn’t it be a disaster if significant changes happen in the product’s UX when the test team is simultaneously testing that particular product flow? It will eventually end up in duplication of work and cause an immense amount of frustration within the test team.
Test automation should be a part of the build-test-deploy pipeline. Whether your project uses the traditional water-flow or kanban model or CI/CD for product development & testing, automation testing should be considered a part of the development.
Automation tests should be integrated into the continuous delivery pipeline, and tests should be executed regularly. Another cause for an automation testing failure is that automation tests are considered necessary during the later stage of product development.
Doing so would lead to the discovery of critical bugs (functional, non-functional, performance, etc.) at a later point in time. This can delay the product release, which can eventually hurt the business of your business.
Automation testing has to be part of the planing and should be included into the SDLC. The more you test, the beter chances of generating the best results out of test automation.
Flawed Planning And Execution
Shifting gears to test automation involves careful planning and sound execution. You need to have a test plan or strategy in place. The test plan should highlight what you plan to achieve through test automation and expected test coverage. There might be other projects in your business where test automation tools were used. While it is important to understand & document those learnings, you cannot follow a one size fits all approach.
There are many open-source tools, and you might be tempted to use those tools since there is no cost factor involved. Before choosing any open-source tool, you need to look at the tool development history, feedback from existing users, and the current state of development. There is no point in using a deprecated tool since the tool will become unusable one day.
Since web browsers are an integral part of any website/web application, you should choose a cloud-based cross browser testing platform like TestCafe. Choosing the right automation tool is the key automation plans. You do not want to regret choosing the wrong or inefficient tool during later stages of testing. Below are some of the things that need to be kept in mind while selecting the automation tool.
What is the kind of throughput when multiple tests are executed in parallel?
Does the automation tool support popular code tracking & bug tracking tools like, Jira, GitHub, Slack, GitLab.
How simple is it to integrate the tool in the CI/CD pipeline?
What frameworks and programming languages are supported by the tool?
What are reporting mechanisms supported by the automation tool and the usefulness of the report’s information?
What sort of tests need to be automated?
What kind of web browsers, operating systems, and devices would be crucial for test automation?
What is the responsibility of every engineer involved in test automation, and what is the kind of skill set required for test suite development?
What programming language and test framework will be used mainly for enabling test automation?
What is the key focus of test automation, i.e., GUI testing, functional testing, Stress testing, etc.?
What kind of CI tools are used for the execution of the builds?
What are the target test coverage and some blueprints/plan to attain the test coverage requirements?
There are many other points that need to be considered, but these will depend on your project’s nature & scale. Once you have the answer to the questions above, you might be able to solve 90% of the problems with automated testing.
Not Doing Maintenance.
If automation testing is implemented correctly, it will allow companies to ship new features or new products to clients much more frequently and with more confidence.
However, most of those benefits rapidly decrease or even goes away if you don’t take the time to maintain your tests.
Maintenance has to be part of your automation suite, unless your codebase never changes you will have to keep on working and maintaining your automation tests, in order for it to keep up with the latest code as you applications evolves. Work that is been done during the software development process, will effect your existing tests in one way or another. Automation engineers might have to adjust existing ones.
Conclusion
Test automation is not rocket science, and the success will largely depend on the test planning & execution. It would be best to avoid automation testing failures discussed above, overlooking these points can result in the loss of the entire test automation task.
Avoiding an automation testing failure is more of the right mindset needed for project planning, development, and testing – the key pillars of any software product.
