Why are Scripting Languages Mostly Faster than Compiled Languages for Executing Automated End-to-End Tests via UI?
Compiled languages are intended for developing system and enterprise applications, rather than being optimized for scripting automated tests
In terms of raw speed, a compiled language such as C# is a lot faster than a scripting language, e.g. Ruby. But I am talking in the context of end-to-end (via UI) test automation. The context matters, an aeroplane goes slower than a Tesla on the ground.
Also, I used the adjective “Mostly” in the context of End-to-End Test Execution. What’s the most common form of test execution a test automation engineer performs every day? Running individual test cases, which is the execution mode of the following activities.
- during creating a new test case
- debugging a test case
- fix a test case ( failed in a run on the CT Server)
- maintain a test case (due to app changes)
- refactor test script
- for creating a data scenario for the team members
- showcase a user story
Test automation engineers typically avoid running an entire suite on their local machines due to the extensive time it consumes. Instead, they execute the complete suite on a Continuous Testing (CT) Server, leveraging parallel build agents. During the suite execution, test scripts coded in compiled languages like Java demonstrate a marginal speed advantage over those scripted in scripting languages like Ruby. (I will elaborate on the ‘slightly faster’ aspect shortly)
My claim (benchmark will be revealed soon) is based on the common end-to-end test execution mode: running an individual automated E2E test case.
For a typical automation test execution, the execution time can be classified into the following categories.
- Compile time
(only required for compiled languages) - Start-up time
(the language start-up time, for example, starting a Java VM takes a while) - Script Step Execution
(the time for the computer to execution the step instruction) - Operation time in App
(such as clicking a button on the payment page, and waiting for receipt) - Complete
(clean up)
For a clearer understanding, I’ve created the following chart.
The chart shows the time blocks to execute a simple 4-step E2E test case. T length of the time blocks is not proportional to reality, just for illustration. The blue part (Script Step Execution) is where rare language speed matters. I enlarged the blue segment in the chart, in reality, it’s so fast that it’s not even visible on the chart.
In terms of raw speed, Java/C# significantly outperforms Ruby. However, as depicted in the chart, the majority of execution time cost is attributed to the app time (shown in red). For example, if the payment process takes 10 seconds, no matter which language you use in e2e test scripts, even C++, the payment step is going to be longer than 10 seconds.
What makes Java/C# slower than Ruby in this E2E execution mode then?
- Compile Time
- JVM or CLR Start-up time
In individual test execution mode, this additional overhead fails to offset the gains made for test step execution (represented by the blue segments in the chart), which results in longer execution time in compiled languages.
Let’s see a benchmark test.
RSpec Test (in Ruby language) — 8.5 seconds
VS Test (C#) — 11.8 seconds
Please note, to minimize the compile time, the project has been compiled previously and with only a simple change to one test script file.
The ruby version (on Gitbub) was about ~30% faster than C#’s (the test steps are exactly the same), in this case.
This is just an overall speed comparison. In a real working situation, the initial delay makes a test automation engineer feel much longer.
To wrap it up, Automated Test Scripts Shall be in the Syntax of a Scripting Language, Naturally! We must refrain from utilizing Java or C# in E2E test scripts; instead, we should opt for a scripting language, and there’s a compelling rationale behind calling them ‘test scripts’. I recommend Ruby, the best Scripting Language for End-to-End Test Automation.
Related reading:
- My “Discover Ruby Programming Through Fun Examples” interactive course on Educative
- Why Scripting Languages are Better for Writing Automated End-to-End Tests?
