Ecommerce Testing: How to combine automated and manual testing for ecommerce stores

Ecommerce testing should be a crucial part of your ecommerce development roadmap, fully integrated with the work of designers and developers.

Running an ecommerce store is never a finished job. As technology evolves, competitors improve and consumer needs change, ecommerce stores need to constantly adapt and evolve the user experience and functionality of the site.

Companies will release and deploy new code to bring some of this new functionality and design to life, often in short bursts, or sprints. But before anything can go live, it has to be tested extensively.

Enter ecommerce testing.

What is ecommerce testing? 

Ecommerce testing is the process of evaluating an ecommerce store to make sure that customers can buy products and carry out all the tasks they need to do. For example reading product descriptions, using filters or comparing products. 

In ecommerce agencies, testing is the main responsibility of the Quality Assurance (QA) team, aka the testing team. The team will work with the strategists and developers to understand the scope of the deployment, write the scripts for automated testing, consider the scenarios and perform all the tests before any new code is deployed. 

That’s not to say the QA team is the only one that tests the site and evaluates its quality. 

Our Solution specialist team will also constantly test the site from the Frontend and Magento admin, to check features are working as expected. In addition, UX Designers will test designs and prototypes with real users during the design process. And of course, developers test all the time as they go.

Types of testing in ecommerce

There are various types of ecommerce testing. They have different scopes, and in some cases are performed by different specialists. 

Functionality testing

When we speak of ecommerce testing, we’re mostly thinking of functionality testing.

Functionality tests verify that users will be able to perform certain tasks and that the features being implemented work as intended. 

These are the tests that might identify that a product can’t be put into a basket. Or that users can’t fill out forms during checkout. Imagine that.  

With functionality testing, it helps to adopt a prioritisation system to label issues. At GENE, we talk about P1, P2, P3 and P4. 

Security testing

Ecommerce Security Testing looks for vulnerabilities of the system and makes sure that the system is protected from possible attacks, intruders or malware. 

Ecommerce sites can often be the victim of cyberattacks, with cybercriminals intent on stealing customer data. 

Keeping your ecommerce platform updated greatly reduces the chances of attacks. This is because criminals tend to focus their attention on sites running on older, less secure versions of ecommerce platforms.  

Having said that, security testing is important in any circumstance. 

Security risks might not be as common as bugs, but they can certainly damage your ecommerce business irreparably. Hence why security testing is a crucial part of our ways of working.

Performance testing

Ecommerce Performance testing is focused on the speed and responsiveness of websites, either in normal circumstances, or to mimic the effect of seasonal peaks. 

Performing regular automated performance testing for uptime and availability, as well as load testing for peak planning (eg, Christmas), can avoid a nightmare at the times when you need your website to be performing at its highest. 

Another key reason for making performance testing part of your process is page load speed. 

The faster the page speed load of your ecommerce store, the higher the conversion rate (as page speed decreases bounce rates on landing), and the more incoming visitors (as page speed increases search engine rankings). 

Usability testing

Usability tests are tests designed to evaluate how many users are likely to be able to conduct certain tasks, and their experience while doing so (eg thoughts and feelings)

Functionality and usability testing are similar in that they both focus on the ability of the user to purchase and conduct tasks. 

However, they have a few key differences. 

Functionality testing is concerned with an objective evaluation of how the online store works (vs how it is supposed to work). 

For example, Can the user put an item on their basket, or can’t they? As in, is it physically possible to do so?

By contrast, a usability test is looking for a more nuanced understanding of the experience of completing a task. Is it intuitive or confusing? Easy or hard to do? Satisfying or boring? 

For example, Do users know how to put an item in their basket, and does it work the way they expected it? And if not, why not? 

In an ecommerce agency, usability tests are done by our Ecommerce experience design team. They seek the understand the average user experience of a user and the level of usability of the site. Then, they improve it with better designs. 

A / B testing (or multivariate testing) 

A / B testing is the process of evaluating how a variant of a design performs with real users in real contexts, against the current design. It is done to find out whether a new variant is more effective at achieving given objectives. 

A / B testing is different to all previous tests in that by definition, it is tested “in the wild”. This makes it very robust testing and a fantastic way to improve metrics such as add-to-basket, conversion rate and average order value. 

A / B testing is done by Conversion Rate Optimisation, Marketing and Design teams to improve the ability of the site to sell, and to learn the effect of different designs on user’s behaviour. 

What is ecommerce automated testing? 

Ecommerce automated testing is the process of evaluating the site’s functionality, security, availability and performance through an automated tool that carries out the QA tasks. 

In automated testing, the QA team will write the scripts that run test scenarios. These scripts will include rules (“always do this when this happens”) and be set on schedules, or even be included in the build scripts. 

The aim of automated testing is to identify issues as quickly as possible, prioritising the evaluation of key consumer journeys (eg. homepage to checkout) so that the biggest issues come to the surface before any human notices them. 

What is ecommerce manual testing? 

Ecommerce manual testing also evaluates the site’s functionality, security, availability and performance, however in this case it is done by the QA team, who will follow a guided process or checklist built based on the scope of the deployment (or whatever the context is) 

What are the benefits of ecommerce automated testing? 

Ecommerce automated testing is a newer form of software testing, which is constantly evolving to become more efficient and fail-proof. 

You won’t be surprised to know that we are huge fans of automation at GENE. This is because there are huge benefits in investing in automated testing tools and processes. 

We are big fans of Cypress.io – check them out if you’re thinking about testing automation or reviewing your tech stack.

This is what you can gain from automation: 

• Speed. We can look at every page type on an ecommerce store in a matter of minutes. 

• Consistency. Tests will look at everything in exactly the same way and order, whereas humans really struggle with that level of repetition. 

• Objectivity. There are no grey areas in automation. A test either passes or fails. There are no instances of “It failed, buuuut it’s kind of okay…”

• Thoroughness. Automated tests don’t get lazy. When was the last time you looked at the footer of a site? A script will never skip a step we have made, so items are never missed.

• They provide data and evidence. We can go back in time and say we ran X test at Y time and here are the results of every single step.

What are the benefits of ecommerce manual testing?  

It’s tempting to have an “in with the new out with the old” attitude. This can be really common in technology companies. 

However, there are benefits to manual testing too! 

Essentially, automated testing should form the foundation of your QA process. Manual testing will then be used to “plug the gaps”. Hold the hands of the machine as it were. 

So what are the benefits of manual testing (provided you also have automated testing workflows?):

• Flexibility: Manual testing provides opportunity to explore and be flexible. Our QA specialists are experts at poking around websites and exploring them in different ways. This allow them to think laterally and be more through with their testing. Also, once your automated scripts are set up, you can use manual testing to test a scenario you forgot to include in the scripts. 

• Easy set-up. Manual testing doesn’t require any set up or maintenance. For simple checks, you might be better off relying on manual testing. 

• Negative testing.  This is the process of deliberately doing things wrong to make sure the website handles them gracefully. For example, clicking buttons twice and making sure we don’t place two orders or checking for field error validation from a usability point of view.