Case Study: Extract All Substack Article Titles and Links. Part B: Extract 25 articles on one page

Add looping.

Non-Medium Members: Read this article free on Substack.

This article series:

• Part A: Extract Individual Article Data
• Part B: Extract 25 articles on one page
• Part C: Extract All
• Part D: Publish
• Part E: Annotation by Zhimin Zhan *
  (offering valuable tips for test automation engineers to level up their skills, exclusively available on Substack)

Continue from Part A. After successfully extracting the title and link of a single article, proceed to retrieve up to 25 articles from a single Substack list page.

Extract all 25 articles on one page

In the special debugging_spec.rb (still in TestWise Debugging mode), change to extract all 25 articles.

article_links.each do |article_link_elem|
    the_data = extract_article_data(article_link_elem)
    File.open("/Users/me/tmp.csv", "a").puts(the_data.inspect)
 end

Please note that I used a (appending flag) when writing to a file, allowing me to view ongoing data, after multiple attempts.