E2E Test Design Anti-Pattern: “The full test script passes, but certain individual tests in it fail”

Understanding and Avoiding Individual Inter-Test Dependency Failures in Automated E2E test scripts

In this E2E Test Design Anti-Pattern series:

• “Test script fails after the first successful run”
• “Each test case, in one test script file, passed on its own but failed when run together”
• “The full test script passes, but certain individual tests in it fail”

This article is based on my father’s new book End-to-End Test Automation Anti-Patterns, but presents simpler (and alternative) examples.

Announcing new book, “End-to-end Test Automation Anti-Patterns”

For the earlier E2E Test Design Anti-Pattern, “Each test case in one script passes individually but fails when run together,” the opposite scenario can also occur. In fact, it’s often a more common anti-pattern among junior-level test automation engineers — “junior” here referring to technical proficiency, not job title or years of experience.

Let’s see an example, a common CRUD test script. (CRUD stands for Create, Read, Update, Delete — the four basic operations for managing a data record)

describe 'Client CRUD' do
    
  before(:all) do
    start_browser
    navigate_to_clients_list
  end

  it "[1] Create Client" do
    # ...  
    
    # Verify the client (READ) after creation
  end

  it "[2] Edit Client" do
    # ...
  end

  it "[3] Delete Client" do
    # ...
  end
 
end

Here are pseudo-script for three test cases:

it "[1] Create Client" do
  click_add_icon
  enter_client_details # name, phone, email, ...
  click_create
  verify_the_new_client # on the client page
end

it "[2] Edit Client" do
  click_edit_icon
  change_client_middle_name
  click_update
  assert_client_full_name_changed
end

it "[3] Delete Client" do
  click_delete_icon
  confirm
  verify_client_name_not_shown_on_clients_list
end

Assuming all test steps are written correctly, a run of the test script will be successful, with ✅✅✅ on those three test cases.

However, if you run the test scripts individually, only the create test will pass. The edit and delete tests will both fail on their own.

Why? The edit and delete client tests rely on a client existing, but the client is only created in the create client test.

How to address this issue? There are multiple ways.

1. Merge individual test cases into one

Not recommended, because each test is testing a different functionality, they are easier to view, maintain and generate reports when kept separate.

2. Setup before test cases

In the above example, simply create a new client beforehand in the edit and delete tests.

it "[2] Edit Client" do
  # create client for editing
  click_add_icon
  enter_client_details # name, phone, email, ...
  click_create

  click_edit_icon
  change_client_middle_name
  click_update
  assert_client_full_name_changed
end

it "[3] Delete Client" do
  # create client for deletion
  click_add_icon
  enter_client_details # name, phone, email, ...
  click_create
  
  click_delete_icon
  confirm
  verify_client_name_not_shown_on_clients_list
end

Create new client in each test where it’s needed instead of relying on a prior test to have created the client.

For those with programming experience you could take this one step further and conditionally only create a client if a client has not already been created.

$client_created = false # flag to check if client has been created yet

it "[1] Create Client" do
  click_add_icon
  enter_client_details # name, phone, email, ...
  click_create
  verify_the_new_client # on the client page
  $client_created = true
end

it "[2] Edit Client" do
  if $client_created == false
    # create client for editing
    click_add_icon
    enter_client_details # name, phone, email, ...
    click_create
  end

  click_edit_icon
  change_client_middle_name
  click_update
  assert_client_full_name_changed
end

3. Ensure test cases are independent by setting preconditions

For more advanced setting-test-precondition, check my father’s article: “My Innovative Solution to Test Automation: ‘The Simpsons’ Data Reset Pattern”.