Benchmark Selenium Test Executions on Different Machines with Different OS and CPUs
Choose a cost-effective build machine for executing automated tests
Execution speed is an important factor in Automated Functional UI Testing, especially for a large regression suite (e.g. 200+ user story level tests, Level 3 of AgileWay CT Grading). While we can run the tests in a Continuous Testing server that supports parallel test execution (like BuildWise), it is still wise to choose a fast, reliable, and affordable build machine for executing automated tests.
In this article, I will benchmark test executions of the same Selenium WebDriver test in the following build machines (in my father’s testing lab):
- MacMini 2012 — i7
- iMac 2015 — i5
- MacMini 2020 — M1 (the base model)
- Windows 10 — in a VM inside iMac 2015
- Linux Mint — in a VM inside iMac 2015
Test Case: a Selenium WebDriver test in RSpec. This is an end-to-end test for WhenWise covering a wide range of web elements in a typical modern web app.
Execution: in TestWise IDE on macOS, Windows, and Linux.
Raw Benchmark Results
- MacMini 2012 — i7
- iMac 2015 — i5
- MacMini 2020 — M1
- Windows 10 — in a VM inside iMac 2015
- Linux Mint — in a VM inside iMac 2015
Comparision
Analysis
- Test executions on virtual machines will be slower than on physical machines. It is not really meaningful to compare the results of these two categories (Windows & Linux above).
- The MacMini M1, to my surprise, is not a clear winner. Maybe it is due to two reasons: Chrome and ChromeDriver are not fully optimized for Apple Silicon CPU yet and TestWise macOS edition is not a native Apple Silicon app (the impact of IDE on test execution shall be minor, though).
Findings
1. The CPU does not matter much
According to cpubenchmark.net, the CPU Marks for MacMini 2012, iMac 2015, and MacMini M1 are 5078, 5658, and 15018, respectively. However, very minor differences were exhibited in the results.
2. The HardDisk speed does not matter much
The NVMe SSD hard drive in MacMini M1 is about 6 times faster than the SSD hard drives in iMac and MacMini 2012. However, the test results on three physical Mac computers are quite similar.
3. Linux is faster than Windows
Windows VM is allocated with 2 CPUs cores and 4.5GB Ram; Linux VM is allocated with 1CPU core and 2GB Ram. Yet, the test execution in Linux is about 13% faster than it in Windows.
The starting browser time in Windows was noticeably slower, compared to macOS and Linux as well.
Recommendation
Based on the benchmark results, I have the following recommendations to keep in mind for build machines (for executing automated tests in Chrome).
- Cheap
No need to buy an expensive computer as a build machine for executing automated UI tests. - Small
So you can fit more in your test lab. - Quiet
It may not sound necessary. However, when your test lab has 5+ machines, you will appreciate the quietness. Also, a quieter machine generally means consuming less power, i.e., cost-saving. - Free OS
Windows 10 is the only consumer OS that is not free. Besides the cost, test execution on Windows is slow. Furthermore, Windows 10 updates can be quite annoying to build agent machines.
In terms of specific machines, my preference is MacMini M1. It is so quiet and decently priced for it’s quality (starting from US$699). FaceBook’s build farm consists of aisles and aisles of Mac Minis. Check it out, and prepare to be impressed!
Linux on a Mini PC, powered by AMZ Ryzen or Intel CPU (such as Intel NUC), is good too. I don’t have a comparable physical machine running on Linux, but I think Selenium test execution on Linux will be slightly faster than it is on macOS.
