A universal approach to ecommerce GUI test automation

Allowing any element of a digital product to fail can have significant implications. Functional failures have tangible consequences for conversion rates, which ultimately impact the entire business. Difficulties with browsing web sites affect user impressions of the brand meaning they may very well leave and never return. This is true for any digital product but is especially the case with eCommerce, where competition is high and any failure can lead directly to customer attrition.

This makes testing of eCommerce sites extremely important. The good news is that modern eCommerce platforms have more or less similar main customer journeys – from account creation to purchase and post-purchase actions. In this article we will show how it’s possible to take advantage of the similarities that occur between various business operations to build a unified solution for testing eCommerce web and mobile sites.

What is the most common issue faced when testing eCommerce platforms? Management are always looking to speed-up changes to be delivered into production and need to be confident that no new major issues will emerge after the update. In order to be able to achieve this in any reasonable time frame, at least some automation of the delivery process must occur and test automation is a significant part of that process.

It is common for businesses to invest in test automation but unfortunately it’s also common for them to run into the following issues:

• Uncertain coverage of test automation.
• Unknown conditions and configurations in which the tests were run.
• Regular failures of automated tests.
• The inability to run tests in a production environment.
• Outdated or missing test data.
• Creating a comprehensive automated regression test suite takes too much time.
• “All tests worked on my machine … yesterday”.

You might be interested to learn that we know how to overcome each of these issues. And our approach is viable regardless of the eCommerce platform size, from giant globally renowned department stores right down to small specialized web and mobile sites.

At Grid Dynamics, we’ve been involved in testing eCommerce platforms from the initial establishment of the company. And we continue to do so today across multiple levels – from experimental distributed backends to customer-visible production interfaces. We have also developed a universal accelerator that enables eCommerce site automation within just a few days. UniECom – remember the name!

Due to the depth of our expertise across eCommerce automation, our teams have been able to develop a stable framework that covers almost all customer journeys. It’s also  versatile enough to address all of the main eCommerce test automation challenges. The principal aim of the solution is to give test automation engineers a powerful tool that can be adapted for the UI elements of all types of eCommerce customers.

The following eCommerce functionalities are included:

• Home page objects
• Customer accounts
• Login and logout
• Product and category search
• Product pages
• Shopping cart pages
• Promotions
• Checkout
• Order purchases
• Gift cards

This results in coverage of more than 120 business scenarios and 10 functional areas.

The following schematic provides a typical end-to-end eCommerce scenario:

To check the possibility of adapting the accelerator for production testing, the QE team used production environments to develop the examples. The experiment’s results showed that it’s possible to apply UniECom for testing on production (while taking all relevant security issues into consideration).

One more factor needs to be mentioned here. While several years ago most purchases were completed directly via the web, today’s eCommerce sites accept a huge range of cross-device and mobile-only orders. Therefore it’s crucial to cover both web and mobile applications with regression testing and pay close attention to the cart merge and other cross-session functionalities.

The following platforms are covered by UniECom:

• Web UI
• Mobile Web UI
• Mobile native applications

So what makes it all possible? Let’s take a look at the dynamics of test creation by a team of two engineers and compare the traditional approach with UniECom’s approach.

As the chart above shows, the pace of test creation is significantly higher in first iterations. This occurs because the testing framework, test data management, and a large number of test scenarios already exist and most of the time is spent on interacting with UI elements (real testing!) as well as integrating with data sources and infrastructure.

Let’s summarize the main features of our accelerator solution and the benefits it can deliver:

Test coverage is known in advance because all test scenarios are already written and addressed for one or more business functionalities. No significant changes in the scenarios are expected.

Initial sets of tests can be run against production. This means they can be used on any low-level environment as well. The approach is CI-ready and provides good step-by-step reporting, which is needed to accurately analyze test results.

Due to the well-thought-out architecture of the testing framework and the decomposition of UI elements and test data from the test steps, it’s easy to support a test suite that is intended to be “green” all the time until an issue appears. Of course, it also requires close collaboration with the development team to reflect changes in the UI or business logic in  a timely manner.

There is no need to wait several weeks for the first fragile tests to appear. You can receive them in just a few days, with all the tests adjusted within a few weeks.

There are several other positive side effects:

• Tests first– if you’re in the process of replatforming you can rely on the existing automated test instead of the formal specifications. This helps to shift test automation left and sometimes even have tests in advance, before implementation is ready.
• Establish a solid test foundation– the solution only provides a suite of basic tests that might not necessarily cover all the specific and unique features of your business. However, this standard set of tests serve as a solid foundation from which it is very straightforward to then extend to more cases. And the standard tests can be used as a basis for writing new tests.
• Cost reductions – the most expensive parts that require strong test automation experts are already implemented. This means the main focus can be shifted to the tests, not the test framework.

The ability of the accelerator to be rapidly adapted for use in new eCommerce customer channels like web and mobile is always a major priority for eCommerce businesses. However, our experience has shown us that eCommerce automation is very often accompanied by a number of challenges that drastically decrease automation speed. Overcoming these challenges was another important goal we had while working on the accelerator.

The following questions are commonly encountered when you start test automation:

• Which tooling should be used and how expensive is it?
• What is the best way to manage test data?
• Which format of tests should be used?
• What reports need to be generated?
• What is the best architecture for the test automation framework?
• How can it be incorporated into the Continuous Integration System and services in test environments?
• How can we ensure the tests are reliable?
• How can the solution be made to be stable and scalable?

Usually the team needs several weeks to be able to answer all of these questions. But unfortunately, the answers reached aren’t always on the mark, or can even be completely incorrect. This is where the UniECom solution shines. It has the right answers and encapsulates a set of best-practices in UI test automation as well as having a well-designed, business oriented test suite out-of-the-box.

Now let’s take a closer look at some of the key QE automation challenges:

The majority of issues can be resolved via integration with the Test Data Management System (TDM). This area will be the subject of our next in-depth article, but let’s touch on some of the key details here.

While working with complex eCommerce platforms you’ll realize that you need to be able to integrate with a number of data sources. And the data there should be in-sync. In addition, the operations you can do with data differ between low-level and top-level environments that require more engineering time to support these differences.

Our approach of managing test data helps to unify and separate the data and test scenarios. Moreover, in the example above we used an intermittent approach, which helps to speed-up test data management even without access to original data sources. It can help to avoid the “I’m blocked” test development approach so that you can continue to work with tests in parallel, while management resolves administrative and organizational issues by accessing the data sources.

UniECom supports multiple “screens” – web browser, mobile, tablet etc. This means the test automation team needs only to worry about UI locators and configuration parameters to make it possible to run the same tests on different screens. Of course, if business flows differ it can require additional “test coding” but maybe this presents the opportunity to ask the important question – “Isn’t that actually a sign that the business flows should all be the same?”

Browsers or mobile devices from cloud farms can also be used without additional effort. Alternatively, an in-house mobile farm can be established. How can this be achieved? Make sure not to miss our article covering this in our blog section in the near future.

In contrast to multipurpose enterprise testing tools, the accelerator is targeted specifically to the eCommerce segment. Enterprise tools also come with significant price and education time requirements so represent a serious investment. By contrast, UniECom is based on open-source software so initial costs are close to zero. However, there is still a learning curve that needs to be considered (miserable if you know BDD, Java, and SQL).

Engineers are always greedy to experiment. They’re interested in new libraries, frameworks, and “why not?” attempts. Curiosity is good and one of the most important characteristics of a test engineer. However, sometimes it’s too expensive for the business. UniECom provides a solid framework that helps to redirect the tester’s curiosity from “how to test” to “what to test”.

eCommerce business is intrinsically dynamic and therefore is very likely to be subject to recurring changes. In addition, typical eCommerce site architecture is complex and commonly integrated with a huge amount of 3rd party solutions and a number of data sources. This ensures that eCommerce automated testing remains a complex challenge with long establishment times and the expectation that problems will be encountered across testing and reporting.

A universal solution that is implemented specifically for eCommerce sites and takes into consideration all the gaps and issues of eCommerce applications provides a clear path to simplifying and speeding up test automation. It additionally provides the benefits of operating from the framework level up and enabling test automation teams to avoid false positive or negative reports.

The UniECom test framework from Grid Dynamics addresses most of the issues discussed in this article and can reduce the initial preparation time from several months down to just a few days. It also opens up the opportunity to put in place test automation capabilities for relatively small retailers. Contact us to learn more and see it in action.