Apr 9, 2022
Set up Running Automated UI tests on your First day at work: Part 5
Setting up a Continuous Testing server to run your tests
Once you have written quite a few functional automated End-to-End UI tests, it is not practical to run all tests on your local machine frequently. Instead, you should run them in a Continuous Testing (CT) server.
You may have heard of Continuous Integration (CI) servers. However, CT servers are different from CI servers. CI servers are more appropriate for executing unit tests, and CT servers are more suitable for executing brittle and longer running functional tests.
What is unique about CT servers?
CT servers provide features that are essential for managing executions of functional tests. CI servers (or no servers at all) lack these features and can lead to frustration down the road when your test suite grows in size. Therefore, I highly recommend using a CT server and setting it up early in your project.
Some of these features are:
- Auto retry failed test execution (once)
Due to many variables (e.g. network, loading time), functional UI tests are flakey by nature. As a result, we do not want to restart the execution if a test fails when it shouldn’t. Instead, we want to rerun that specific test! This feature is a huge time saver. - Manual retry
Sometimes, auto retry may not be enough. For example, an external service is down. In such cases, manual retry helps to reduce false alarms. - Intelligent ordering
This feature changes the run order based on a variety of factors. This can ensure quick feedback as new tests, recently failed, and commonly failed tests are run first. Some CT servers even allow you to edit the priority of tests manually so you can push a particular test to run earlier.
Installing BuildWise
BuildWise is a free and open-source CT server. It is quite easy to install, around under 10 minutes. For instructions to install BuildWise, please follow the guide in the link below up to the Create a Build Projectsection.
You should now have BuildWise running on your http://localhost:3618 .
Then log in (default username admin and password buildwise).
BuildWise — How does it work?
All CI/CT servers (including BuildWise) at a high level perform the following core tasks:
- Pull the latest commit from the source
- Execute your tests
- Show test results and execution history
- Reporting
Setup a Sample Project in BuildWise
First, let’s clone a sample project. Make sure you have Git installed.
On your command line, navigate to a working folder of your choice and clone the BuildWise provided sample project.
git clone https://github.com/testwisely/buildwise-samples.gitOnce complete, you should see this output on the command line:
C:\Users\CIO\Documents\Courtney>git clone https://github.com/testwisely/buildwise-samples.git
Cloning into 'buildwise-samples'...
remote: Enumerating objects: 223, done.
remote: Counting objects: 100% (223/223), done.
remote: Compressing objects: 100% (153/153), done.
remote: Total 223 (delta 75), reused 201 (delta 56), pack-reused 0R
/s
Receiving objects: 100% (223/223), 3.66 MiB | 5.75 MiB/s, done.
Resolving deltas: 100% (75/75), done.Open http://localhost:3618/ in your browser and click the ‘Create a New project’ button.
On this screen use the ‘Fill demo project’ drop down to prefill some fields for you.
In this guide, I am selecting the RSpec demo, but the sample Git repository supports all listed options.
In the ‘Working folder’ field, make sure to change this to the actual location of your cloned repository. Mine was in C:\Users\CIO\Documents\Courtney\buildwise-samples.
To create your project, click ‘Create’.
Back at the home screen, there should be a new project.
Trigger a build (execute all the tests) by clicking ‘Build Now’.
You should see windows open and the tests start to run on a demo site. You will also see the test results appearing on BuildWise (browser).
Once the build completes, you will see a summary test report.
At the top (highlighted in orange), you can see the number of failures overall. It is colour coded (green if all passed or red if any failures).
The greenbox shows the details of each test file and case within the files. Here the passenger_spec.rb failed. By clicking the camera button next to a failed test (highlighted in red), you can open a detailed error view.
This error view contains a screenshot taken when the test failed, and the error message. It also allows you to copy the line of the error so you can use an editor to navigate to it.
Hopefully you can see that it is very easy to set up your project in BuildWise. One-click execution and detailed error messages have helped me to run and detect errors.
From now on, you can click one button to run all your tests. You will get feedback quickly and are free to do other tasks while the tests are executing.
For more on CT, check out this book, Practical Continuous Testing.
