Visual Regression Testing

Quality assurance is the most important part of delivering the products. There are many kinds of testing that can be carried out to assure the quality of the product. One of them that helps to find the visual defects which can't be found through bare eyes in the web pages is Visual testing. A few of the visual change examples are change in font size, change in space, change in colour, change in alignments, etc.,

Why did we implement it?

As we are doing subsequent releases every week, there may be a chance to miss the visual and cosmetic bugs that may affect the user experience. So, we thought the visual testing would assure the quality of the components before releasing to the production. In addition, this's helpful for developers as well as testers to find the bugs in different phases of component development.

Selection of tool/library

Our proof of concept for tool selection started here. Once we decided to implement this visual testing to our scope, we were searching for the tool/library which gives us an effective way of testing the visual bugs. We've come across tools like wraith, PhantomCSS, backstop js, etc ., Eventually, we decided to go with backstop js based on the few deciding factors like cost, maintenance, accuracy, and implementation, etc., It's a free source and we can add scenarios and do maintenance easily. In addition, it gave us better accuracy in finding visual defects and this could be implemented easily with other frameworks as well. So at last our chosen library is backstop js.

How backstop js works

Backstop js is a javascript library used to compare the web UI/components between two different environments in various viewports and provide visual changes. We used to compare our components between the testing environment and the reference environment. So this library takes the screenshots of all the scenarios(each component is a scenario) in different viewports in two different environments, compares those snaps and provides us with the result.

How we integrated backstop js

Backstop js can be installed through npm. So we added it as a dependency in our testing framework and utilize it. Backstop config is the file where we mention our test scenarios and viewports details. This is the core of backstop js. All information to run the tests will be fetched from this file. The file contains objects scenarios and viewports and paths to the screenshots and report.

How to add and execute the scenarios

Multiple scenarios can be added as an array of JSON in object scenarios which has the different name/value pairs to run each scenario. In the same way, viewports are also added as an array in the viewports object. Below are the mentioned scenarios and viewports formats used in a backstop config file.

Scenarios

"scenarios": [
    {
      "label": "Name of the scenario",
      "cookiePath": "backstop_data/engine_scripts/cookies.json",
      "url": "https://testingURL.com",
      "referenceUrl": "https://referenceURL.com",
      "readyEvent": "",
      "readySelector": "",
      "delay": 1000,
      "hideSelectors": [],
      "removeSelectors": [],
      "hoverSelector": "",
      "clickSelector": "",
      "postInteractionWait": 0,
      "selectors": [],
      "selectorExpansion": true,
      "expect": 0,
      "misMatchThreshold" : 0.1,
      "requireSameDimensions": true
    }
  ],

Viewports

"viewports": [
    {
      "label": "phone",
      "width": 320,
      "height": 480
    },
    {
      "label": "tablet",
      "width": 720,
      "height": 768
    }
]

We added scenarios for each component by using the URL of the components from the live and testing environments and also viewports we needed to test our components. In each scenario, referenceUrl, url and delay are the most used data.referenceUrl refers to the URL of the component in its environment that is treated as reference (eg. live environment) and url refers to the URL of the component that needs to be compared with reference URL.

Once scenarios are added, we should trigger the CLI commands to get the final result. These CLI commands can be customised if we want. In backstop js, backstop reference is the command used to create the screenshots of all reference URLs in each scenario and backstop test command is used to create the screenshots of all test URLs in each scenario and compare all reference and test screenshots. After backstop test is completed, it'll generate the HTML report and open it in default browser.

How we customised tests to make efficient

Adding lots of scenarios in config file reduced the performance of the tests. So we decided to capture all the HTML combinations of each component in separate files and make those files dynamically added to the config file. It also reduced the manual effort of replacing the URLs for each scenario every time. In addition to that, we changed CLI commands to run the tests in different test environments.

Final report of the tests

We would wish to get a positive output every time for our actions. But here we can't expect that always. Yes, the report of the visual tests. Backstop js gives a very clear and accessible test report to check the passed/failed test scenarios and to compare the difference. It has the scrubber used to compare visual changes between the reference and test images by dragging it.

Below is the report of the tests where we could see the passed scenarios. REFERENCE is the column in the report where it displays all the reference URL screenshots and TEST is the column where all test URL screenshots display. If there are any visual changes between reference and test, DIFF column will be generated in the report where we could find the changes.

How visual tests benefit us

It saves a lot of time during the testing phase of each release and we found many visual bugs using backstop visual tests. After the customization we did, it only took 5 mins for executing all the components in four different viewports. The visual defects raised have been drastically reduced after this implementation.

Future improvements in the pipeline

We have the below plans to enhance the testing scope.

• Implement the same level of approach in products teams to test their corresponding web pages.
• Include the visual testing in CI/CD pipeline so that it'll be triggered for every commit in the pull request