Managing QA Complexity

The Benefits of an Integrated Test Automation Approach

Key Takeaways

New trends in the software industry

Software, and subsequently quality assurance, has largely been deterministic – the output could be precisely determined based on the input. However, the increased adoption of machine learning and AI in the past couple of years has made the software industry much more dynamic.

The last edition of the World Quality Report has identified three key trends in the software industry with a potential impact potential on quality assurance, which I will briefly discuss below.

The growth of the digital economy

The first trend relates to the growth of the digital economy. It puts software at the heart of almost everything any company does. More and more companies from unrelated business sectors have begun to digitalize their business models and to offer more personalized content to their customers. Add disruptive competition from start-ups to this picture, and you can get an idea of what success in the digital economy boils down to. The speed and the quality in delivering software products have become the main differentiators for businesses competing in a fast-paced marketplace.

Rapid adoption of Agile and DevOps

The second trend refers to the agile collaboration between the development and operations teams. This fast-paced collaboration holds the promise of shorter release cycles and increased time-to-market. ‘How short can that time be?’, you may ask. The answer is 11.6 seconds. This is how fast Amazon deploys new software to production. No wonder that the number of organizations that use or experiment with DevOps principles has steadily grown from 7% in 2015 to 47% in 2017, the World Quality Report states.

Big Data and Machine Learning on the Rise

Last but not least, data creation is currently skyrocketing thanks to fast paced innovations in technology and connectivity, giving companies access to an ever-growing stream of insight-rich information. According to experts’ estimations, the data we create is likely to double in size every two years in the next decade. An autonomous car alone will be responsible for generating 4,000 GB a day.

Machine learning algorithms slowly gain momentum and give way for non-deterministic software. The services that use this kind of algorithms generate a range of correct outputs rather than precisely determined ones, which makes the task of testing a single correct behavior more complicated. Here is where AI and test automation come into play and help organizations deliver better quality to their customers.

Benefits of Test Automation in Quality Assurance

As it is well known, quality assurance represents a very important activity in the development lifecycle of software - automating it is even more rife with benefits. However, finding time and resources to test an application is often a challenge for many organizations, and DevOps is adding pressure to that trend. Since almost ¾ of the test activities are repetitive, test automation is better suited to execute these tests. Leaving the heavy-lifting to machines would mean empowering testers to use their reasoning ability more, and reduce the inaccuracies of intensive manual testing.

Test automation is key in the world of Agile and DevOps, but very few companies fully engage in coordinated automation activities. According to the latest World Quality Report, a significant number of companies who took part in the survey reported - among other topics - that they have difficulties integrating test automation into a DevOps process (38%) or integrating the different automation tools together (39%).

Nevertheless, these difficulties are heavily outweighed by the benefits of implementing automation in Quality Assurance, a few of which I will quickly present below.

Cost reduction

Manual testing can be overwhelming when one needs to deploy every 11.6 seconds. Many companies are yet reluctant to implement test automation because of the initial investment. The cost of fixing defects is one of the largest in software development and it increases with each development stage closer to production. Finding a defect in production is significantly more costly to fix than one found in development. It is needless to speak about the customer dissatisfaction, reputational damage and the potential loss of business opportunities. This is why test automation assists in finding defects much sooner, and thus heavily reducing costs.

Safeguard against regressions

Regressions may be slow and find less issues if they are performed manually. Using test automation for regressions allows a more consistent environment because the procedure is stable across time and does not depend on a person. Automated regressions may be run cheaper and more often. Last but not least, the execution is better documented and is available at any time.

Support continuous integration

CI/CD (Continuous Integration & Continuous Deployment) means that constantly the latest version of software is compiled, built and deployed. This cannot be done without test automation. Most of the changes that are committed via the CI/CD pipeline are at a smaller scale. Performing manual tests for each change can be very time consuming and prone to errors. Integrating test automation with a CI server ensures that the team can focus on things that humans are better at rather than on repetitive tasks such as testing.

Expanded test coverage across devices and browsers

Given today’s challenges in the world of software, a sufficient test coverage comes together with great effort. To make sure that an application behaves consistently across devices and browsers, a great number of test cases are required. A robust test automation framework can easily scale a test case with a minimum amount of effort.

Faster time-to-market

Test automation can be parallelized across multiple servers and executed even during off-times. The tests are run faster compared to manual testing, which gives the team quicker feedback regarding the quality and the availability for release. Planning the test automation earlier means finding defects earlier and delivering faster.

What we are doing

Having understood all the benefits involved, Netcentric rolls out AI-based test tools and test automation routinely. Depending on the project specifics, we currently employ a mix of OOTB and custom automation frameworks.

AI-based testing with Applitools

ApplitoolsEyes is a cloud-based testing solution designed to search for visual differences just like a human eye would. Thanks to an intelligent machine learning algorithm, Applitools is able to spot changes and rank them based on the probability of whether they are bugs or new features.

Our needs exceeded the standard functionality offered by Applitools, and therefore we had to make some enhancements.

Our Solution

The ApplitoolsEyes AI engine together with our Selenium Grid both help us scale the power of AI to multiple test environments. Using Google Sheets allow us to be fully flexible and easily maintain test repositories in case of redesign. We thus obtain a better test coverage by simultaneously supporting multiple browsers and viewports.

Integrated test automation with AEM

As an Adobe Premier Solution Partner, we are leveraging the Hobbes.Js framework for testing the AEM applications. This JavaScript based solution can be executed in a real browser in any environment and has all the basic test methods and assertions available. Some operations as well as interface inconsistencies were harder to automate with the out of the box solutions. We therefore developed our own.

Our Solution

Our Hobbes.Runner better supports DevOps by working seamlessly with a continuous integration server such as Jenkins. You can either choose to run it locally or to use the SeleniumGrid to scale your test on several environments. We have also expanded the coverage of AEM Author to include AEM Publish as well, and made it more flexible.

Additionally, it can integrate with Slack, the collaborative chat system we use. As a proof of concept we have build a integration into Amazon’s Alexa to be able to launch tests, and check results through voice commands. Starting automated tests and getting information about the results has never been easier.

Conclusion

Being agile and engaging in DevOps without test automation is doomed to fail in the long run. Operating an integrated test environment where automation and AI-based tools work together is key to a successful digital transformation. Of course, machine-learning-led QA may not take over immediately. However test automation will play an increasingly important role in the testing activities.

A synergy between human testers who excel at creativity on one hand, and flexible frameworks that are scalable and adaptable on the other is a strategic enabler that each company should start considering. Netcentric has fully understood this dynamic and is ready to provide tailor-made services to all our customers who operate at the forefront of innovation.

tag
Developer Circle
style
dark-plum