E2E Test Design Anti-Pattern: “Test script fails after the first successful run”

Understanding and Avoiding State-Dependent Failures in Automated E2E test scripts

In this E2E Test Design Anti-Pattern series:

• “Test script fails after the first successful run”
• “Each test case, in one test script file, passed on its own but failed when run together” (upcoming, check it out next weekend)
• “The full test script passes, but certain individual tests in it fail” (upcoming)

This article is based on my father’s new book End-to-End Test Automation Anti-Patterns, but presents simpler (and alternative) examples.

Announcing new book, “End-to-end Test Automation Anti-Patterns”

Let’s see this simple create a new client test script (pseudo-script shown below), created by Tonpa, a test automation engineer.

describe "Client CRUD tests" do
  
  before(:all) do
    start_browser
    navigate_to_clients_list
  end 

  it "Create a new Client"
    click_new_icon
    enter_first_name "James"
    enter_last_name "Bond"
    enter_email("james@007.com")
    click_create
    verify "James Bond" appear in the client list
  end
  
end

Very simple, right? Assuming every test step is written correctly, it passes. Tonpa is satisfied and reports to the manager that the task is done.

The manager happens to be a curious one, decides to run the test script on his own machine. The test failed with an error: “The email ‘james@007.com’ is already in use”.

When Tonpa received the feedback, his first reaction was surprise: “It passed fine on my machine!” He reran the test — and sure enough, it failed exactly as the manager had reported.

Most readers would probably know the cause — it’s obvious in this example — and might even think, “Tonpa’s skills aren’t up to par.” The reality is, majority test automation engineers also struggle with this when faced with slightly more complex cases (still the same anti-pattern) at work.

During my internship, I observed this anti-pattern in the work of a senior test automation engineer — the lead of the ‘Test Excellence Centre’ at a large telecom company. Of course, their small suite, consisting of just a few dozen E2E tests, had been failing (for a long time). I discovered this after repeatedly asking them for Playwright examples, which they eventually disclosed unwillingly.

My father created a simple interview test-design-scenario question based on this anti-pattern. Not a single candidate answered it correctly, for years! When he revealed the correct answer, no one could dispute its validity. In one instance, a recruitment agent angrily came and confronted him about the results of three candidates he had sent (sequentially). After my father explained the question and the answers from the three candidates, the recruitment agent quietly walked away.

One Simple Test Automation Scenario Interview Question that Most Candidates Failed;

Back to the Tonpa’s case. He deleted the client “James Bond” entry in the database, rerun the test, it passed. Of course, that’s not the solution, for E2E test automation.

What is the solution then?

The principle at play here is test data management — more specifically, managing test data for test automation. This is rather an in-depth topic. While the overarching goal is to keep test data reusable, in practice, the implementation strategy can vary depending on the specific test cases and the application under test.

For this Create-Client test, the simplest form of this anti-pattern presents, the best solution is to change the test design: create a unique client each time.

it "Create a new Client"
    click_new_icon
 
    new_client_first_name = Faker::Name.first_name
    new_client_last_name = Faker::Name.last_name
    new_client_last_email = Faker::Internet.email
 
    enter_first_name new_client_first_name
    enter_last_name  new_client_last_name
    enter_email new_client_last_email
    click_create
    verify new_client_first_name + new_client_last_name appear in the client list
  end

I used Faker, a test data generation library, which is available in most major programming languages.

Why this Anti-Pattern is rather common?

From a technical standpoint, solving this — at least for simple business scenarios — is not difficult. In my view, the key issue is mindset. Many automated testers enter the field through record-and-playback tools or similar utilities, and as a result, they often lack the habit of thoughtfully crafting their tests, using programming skills.

Related reading:

• My book: Web Test Automation in Action: Volume 1
• My father’s new book: “End-to-end Test Automation Anti-Patterns”

The Resources Zhimin Created to Help Achieve Real End-to-End Test Automation & Continuous Testing