Automated Testing Elements on a Lazy Load Page with Selenium WebDriver

How to verify elements that get dynamically loaded on a page using Selenium WebDriver

Some modern websites now use lazy loading (also known as progressive loading). Instead of loading everything on the page at once, the items load as you scroll. This brings challenges to test automation, because the element we want to control may not be displayed yet. An example site is Substack, if you scroll down on this site, you will notice the bottom of this page reloads more articles.

This article will show you how to use Selenium WebDriver to force lazy-loaded pages to load your desired element, and drive it.

Test Design

1. Look for the element on the page
2. If the element is not there, scroll down the page
3. Repeat steps 1–2

1. Look for the element on the page

First, try to locate the element. In the best case, the element is already there on the initial page load. This means there is nothing to do, and we can proceed normally.

driver.get("https://agileway.substack.com/archive")
driver.find_element(:link_text, "12 Payment Test (AJAX)").click
try_for(4) { expect(page_text).to include("Selenium Explicit Waits") }

For this article, I will be trying to click on the article “Fake End-to-End Test Automation Clarified” from the Substack archive list. This was published a while back, so if we try to do this straight away, there will be a NoSuchElement error.

driver.get("https://agileway.substack.com/archive")
driver.find_element(:link_text, "Fake End-to-End Test Automation Clarified").click

Below is a failed test execution in TestWise.

2. Scroll down if the element is not there

Because the element isn’t there yet, keep scrolling down the page until it is. The easiest way to do this is to send the “Page Down” key to the browser.

driver.find_element(:tag_name, "body").send_keys(:page_down)

We don’t know how many times we have to scroll down, so for now, I will use an arbitrarily large number, 100. But set a condition to exit if the desired element is found.

100.times do |x|
    puts("Scroll: #{x + 1}")
    driver.find_element(:tag_name, "body").send_keys(:page_down)
    sleep 1
    begin
        driver.find_element(:link_text, "Fake End-to-End Test Automation Clarified").click
        # quit the loop if found
        break 
    rescue => e
        # not found, continue
    end
end

The above code block:

• Starts a loop that stops at 100
• Sends the “Page Down” key to the browser
• Try to click a specific article.
• If it can find the article, quit the loop (break).
• If it can’t find the article, continue the loop.

After the loop completes, assert that we are now on the article’s page.

expect(page_text).to include("After knowing what is not real, motivated professionals can learn and achieve the real.")

Below is a successful test execution in TestWise, scrolled 14 times.

However, the “Page Down” key doesn’t take us to the bottom of the page and trigger the lazy load. To be more efficient, we should go directly to the bottom of the page.

Alternative Approach

In SubStack, there is a div that indicates the “bottom” of the currently loaded page (lazy loaded elements get loaded before this div).

In Selenium, we can use an advanced user interaction move_to to jump directly to this element, specifically, move the mouse to that last element <div class="visibility-check">. This action is equivalent to page scroll-down in this case.

100.times do |x|
    puts("Scroll: #{x + 1}")
    
    # scroll to the visibility-check element at the bottom of the page
    elem = driver.find_element(:class, "visibility-check")
    driver.action.move_to(elem).perform
    sleep 1

    begin
        driver.find_element(:link_text, "Fake End-to-End Test Automation Clarified").click
        break
    rescue => e
        # not found, continue
    end
end

sleep 3
expect(page_text).to include("After knowing what is not real, motivated professionals can learn and achieve the real.")