Test Your Web App on 500+ Browsers? Tips for Automated Smoke Tests Using Sauce Labs

April 18 2017
 

Test your web app on 500+ browsers by learning more about the gotchas that were discovered using Sauce Labs, and how to deal with them.


It goes without saying the goal of adding functionality is to add a new feature without it disrupting everything else. It’s also nice when you fix a bug without it breaking the system. But, unfortunately most of us have encountered that awkward moment when a new check-in broke the functionality of our app. Don’t feel bad. It happens a lot.

Luckily we have regression tests to help us out, but in the business of dealing with browsers and end users, how do you deal with the complexity of over 500 combinations of operating system and browser varieties?

At AppDynamics, we use Sauce Labs and Selenium in our automated smoke tests to help do just that. Sauce Labs provides test automation across virtual machines of browsers and platforms. With Sauce Labs’ Platform Configurator, you first specify the API, device, OS, and browsers, then you’re a button click away from configuring the desired capabilities of your test.

Depending on your infrastructure and test environment, you can require the browser to have access to a server inside the firewall. Sauce Connect opens a channel between your local network and the remote browser in the testing environment in Sauce Labs, so the VM can access the test system’s server.

All the above is standard information, and you can find more details about using Sauce Labs from their Wiki. So let’s dig in and discuss some of the gotchas that we discovered using Sauce Labs, and how we deal with them.

Some challenges found when working with Sauce labs

1. Cache busting

Although Sauce Labs spins off pristine browsers for each test session, the browser can take a significant amount of time to start up. To amortize these costs, we reuse a single browser session for all the tests. Cached website data can skew the test results. If you want to get metrics such as page load time and DOM first byte ready, you need to make sure the resource under testing is not cached. When you instruct WebDriver to visit a URL, you need to consider the usage of some cache busting techniques, such as cache control headers and unique fragments. This way ensures a unique URL for each visit, and no confusing results will be generated.

This code is in Java with Selenium. We added UUID to the end of each URL request.

void loadURLInBrowser(String url, boolean reload) {

       UUID uuid = UUID.randomUUID();

       String hashString = "#" + uuid;

       browser.get(server.absoluteUrl(url  + hashString));

       server.beaconSaver.waitForExpectedEvents(beaconCount);

       if (reload) {

           browser.navigate().refresh();

       }

}

2. Unknown_ca errors when testing HTTPS

Browsers don’t send HTTPS request if they can’t find the right SSL Certificate. When the server uses a self-signed certificate (for the convenience of tests), the Sauce Labs browsers don’t trust this certificate. Instead, they modify the browser to trust their own certificate, which is installed in their VPN proxies. Newer versions of Safari and some other browsers are more secure and harder to modify, so they throw this error. Sauce Labs knows this issue, but hasn’t spent the time to figure out how to modify it. This creates problems, and you’ll see unknown_ca errors.

In our testing, we had this problem with:

  • Android 4.4 and 5.0
  • iOS 7.1 and later
  • Safari 8 and later
  • Edge

You have to avoid tests that depend on the Sauce Connect proxy for HTTPS on these browsers.

3. Clock synchronization issues between your local server and remote web browsers

Performance monitoring collects lots of timing metrics data, and we often need to validate a timestamp in our tests. However, the clock on the remote machine may not be synchronized with the clock on the local machine. The timestamps may differ significantly. When you design your tests, you need to keep the clock synchronization issue in mind. Likewise, network latency can be an issue. Requests and responses can arrive out of order, even when you try to add latency to responses. It’s best to skip tests that depend on the order of asynchronous requests.

4. Be sure to check a port’s availability

If your test requires a distinct web server to go with each test, then you need to make sure the server is running and has the port open before you send the address to the browser. When starting tests in parallel in threads, the server could choose a port and report it to the browser before the port is opened, and it could fail because another server could open that port first.

5. Problematic tunnel ports: 6000 and 6666

If your pages load from localhost, then to be compatible with all browsers you need to use the ports tunneled by the Sauce Connect proxy in the VM, in accordance with the FAQ at Sauce Labs (https://wiki.saucelabs.com/display/DOCS/Sauce+Connect+Proxy+FAQS). However, we’ve empirically had issues with ports 6000 and 6666, so we removed those from our list. We also grouped the ports into pairs, so we could do HTTP/HTTPS tests.

public class SauceInfo {

   static final int[][] PORTS_TUNNELED_BY_SAUCE_CONNECT = {

           /* 6000 Safari doesn't want to connect to this, even when it's available (!?),*/

            /* 6666 Safari doesn't like ,*/

           {80, 443}, {2000, 2001}, {2020, 2109}, {2222, 2310}, {3000, 3001}, {3030, 3210}, {3333, 4000},

           {4001, 4040}, {4321, 4502}, {4503, 4567}, {5000, 5001}, {5050, 5432}, {6001, 6060}, {6543, 7000},

           {7070, 7774}, {7777, 8000}, {8001, 8003}, {8031, 8081}, {8765, 8777}, {8888, 9000}, {9001, 9080},

           {9090, 9876}, {9877, 9999}, {49221, 55001}

   };

}

Hope this article helps. Have fun playing with Sauce Labs in the smoke tests of your web apps.

 

Nicole Hu

Thank you! Your submission has been received!

Oops! Something went wrong while submitting the form