Case Study: Image Manipulation in Automated End-to-End Tests

Automate an iOS Tic Tac Toe app with Appium 2 (XCUITest) Part 2: Player vs Computer.

In a previous article, I walked through using Appium 2 to play a TicTacToe game (on iOS). That was Player vs Player mode, i.e. we can plan the moves to get desired output.

I will add a more dynamic twist in this article by choosing Player vs Computer mode. Please check the previous article’s set-up guide if you want to follow along.

Test Case:

To complete a Tic Tac Toe game (on iOS) in the Player vs Computer mode using automated end-to-end test scripts.

This app has 9 cells, each can be one of three states:

• blank
• ✕ image
• ⭕ image

The challenge of this test case is to tap an available (i.e. blank) cell, which are non-deterministic.

In Appium, to my knowledge, there is no way to retrieve the image’s name. Therefore, we can’t detect what is in the cell. This translates to an image detection problem.

The test script for this article is in Ruby, the best language for scripting automated tests. There are intuitive image manipulation Ruby libraries (called gems), which I will use to accomplish the task.

Test Design

1. Take a screenshot of the app after each move, and save it to a file.

2. Crop the board (square shape) out of the screenshot.

3. Crop the board to generate nine equal-sized images (to represent the cells).

4. Analyze each cell image to determine whether it is blank or not.
The easiest way to do it is via colour detection. Please note, whilst it appears just red or blue to human eyes, there are many more colours (anti-aliasing). We can use the number of unique colours to differentiate blank and played cells.

5. On the player’s turn, place a token in a random blank cell until the game ends.

Preparation

Install required gems.

1. ImageScience

This is an image cropping library.

% brew install FreeImage

% gem install — no-document image_science

2. ChunkyPNG

This gem gets PNG info, including each cell’s colours.

% gem install-no-document chunky_png

Steps

1. Save a screenshot of the app in Appium.

Appium has a built-in function screenshot-as which saves a screenshot of the iOS screen.

# screenshot then save as PNG in /tmp/base64.png
png_base64 = driver.screenshot_as(:base64)
File.open("/tmp/base64.png", "wb").puts(Base64.decode64(png_base64))

The screenshot is encoded as Base64 format. To save it as a PNG file, decode the Base64 and save it in binary file format (with wb).

Example base64.png screenshot

2. Crop the board (square shape) out of the screenshot.

Knowing the exact area of where to crop the board is finicky. I had to manually open the saved image and figure out where precisely the board began and ended, as well as the board’s dimensions.

The board was 1074 x 1074 pixels. The board offsets were 48 pixels (from the left) and 688 pixels (from the top). With this, we can crop the board.

width = 1074
height = 1074
#  cut images with ImageScience library
ImageScience.with_image("/tmp/base64.png") do |img|
    img.with_crop(48, 688, 48 + width, 688 + height) do |crop|
        # save the board to /tmp/board.png
        crop.save "/tmp/board.png"
    end
end

Example board.png

3. Crop the board to generate nine equal-sized images.

In this step, we take the previously generated board.png and slice it into 9 equal squares.

cell_width = width / 3
cell_height = height / 3

ImageScience.with_image("/tmp/board.png") do |img|
  cell_list.each do |x|
    row = x / 3
    col = x % 3
    img.with_crop(col * cell_width, row * cell_height, (col + 1) * cell_width, (row + 1) * cell_height) do |crop|
      # save each cell and name the file with the row-column coordinates
      crop.save "/tmp/cell_#{row}_#{col}.png"
    end
  end
end

Example cell_0_0.png (upper-left-most cell)

4. Analyze each cell image to determine whether it is blank or not?

To check if a cell is blank, I used ChunkyPNG to returns the number of unique colours present. If the number is low (<1000), then we can assume the cell is blank.

Because of the red cell border and anti-aliasing, there are around 200 colours for a blank cell. I’ve set the limit to 1000 just to be safe.

def is_blank_cell(img)
    height = img.dimension.height
    width = img.dimension.width
    puts "height: #{height}, width: #{width}"
    color_codes = []
    height.times do |i|
      width.times do |j|
        arr = [ChunkyPNG::Color.r(img[j, i]), ChunkyPNG::Color.g(img[j, i]), ChunkyPNG::Color.b(img[j, i])]
        color_codes << "\##{arr.map { |x| x.to_s(16).rjust(2, "0") }.join.upcase}"
      end
    end
    color_codes.uniq!
    return color_codes.size < 1000
end

5. On the player’s turn, place a token in a random blank cell until the game is ended.

As shown in the previous article, playing a cell is as simple as tapping it. The problem in this step is choosing a random blank cell.

I decided to record all the blank cells in an array named blank_cells. I used the sample function to randomly select one blank cell, as shown below. A bit of programming required here.

blank_cells = [0, 1, 2, 3, 4, 5, 6, 7, 8]

# save and crop the board
blank_cells.each do |x|
    row = x / 3 # convert index to coordinates
    col = x % 3
    img = ChunkyPNG::Image.from_file("/tmp/cell_#{row}_#{col}.png")
    # remove all cells that are not blank
    blank_cells.delete(x) unless is_blank_cell(img)
end

# randomly select one of the blank cells and play it
empty_cell = blank_cells.sample
board[empty_cell].click

The above code block only does one turn. To play a full game, we should add a loop so that the script keeps going until the game finishes.

6. Determine whether a game is completed or not.

The app has a game status text, it’s two states are “Game is playing” or “Game has ended!”

Game in progress text (on Computer’s turn)

Game completed text

We can loop the player’s turn based on the value of the game status text.

blank_cells = [0, 1, 2, 3, 4, 5, 6, 7, 8]

while blank_cells.count > 0 && driver.find_element(:class_name, "XCUIElementTypeStaticText")["value"].include?("Game is playing")
  sleep 0.5 # wait for computer's turn to finish
  if blank_cells.size < 9 # if all blank = 9, the first move, don't need check
    save_board_screenshot
    smart_crop_board_into_9_images(blank_cells)

    blank_cells.each do |x|
      row = x / 3
      col = x % 3
      img = ChunkyPNG::Image.from_file("/tmp/cell_#{row}_#{col}.png")
      blank_cells.delete(x) unless is_blank_cell(img)
    end
  end

  puts blank_cells.inspect
  empty_cell = blank_cells.sample
  board[empty_cell].click
  blank_cells.delete(empty_cell)
end

expect(driver.find_element(:class_name, "XCUIElementTypeStaticText")["value"]).to include("Game has ended!")

Video Execution

Animated GIF

YouTube video.

Complete Test Script

load File.dirname(__FILE__) + "/../test_helper.rb"
require "base64"
require "chunky_png"
# brew install FreeImage, then gem install image_science, https://github.com/seattlerb/image_science
require "image_science"

describe "Player Vs Computer Games" do
  include TestHelper

  before(:all) do
    @driver = Appium::Driver.new(appium_opts).start_driver
  end

  after(:all) do
    driver.quit unless debugging?
  end

  def smart_crop_board_into_9_images(cell_list)
    width = 1074
    height = 1074
    #  cut images with ImageScience library
    ImageScience.with_image("/tmp/base64.png") do |img|
      img.with_crop(48, 688, 48 + width, 688 + height) do |crop|
        crop.save "/tmp/board.png"
      end
    end

    cell_width = width / 3
    cell_height = height / 3

    ImageScience.with_image("/tmp/board.png") do |img|
      cell_list.each do |x|
        row = x / 3
        col = x % 3
        img.with_crop(col * cell_width, row * cell_height, (col + 1) * cell_width, (row + 1) * cell_height) do |crop|
          crop.save "/tmp/cell_#{row}_#{col}.png"
        end
      end
    end
  end

  def save_board_screenshot
    png_base64 = driver.screenshot_as(:base64)
    File.open("/tmp/base64.png", "wb").puts(Base64.decode64(png_base64))
  end

  def is_blank_cell(img)
    height = img.dimension.height
    width = img.dimension.width
    puts "height: #{height}, width: #{width}"
    color_codes = []
    height.times do |i|
      width.times do |j|
        arr = [ChunkyPNG::Color.r(img[j, i]), ChunkyPNG::Color.g(img[j, i]), ChunkyPNG::Color.b(img[j, i])]
        color_codes << "\##{arr.map { |x| x.to_s(16).rjust(2, "0") }.join.upcase}"
      end
    end
    color_codes.uniq!
    return color_codes.size < 1000
  end

  it "Player vs Computer, Taking screenshot" do
    driver.find_element(:name, "Player VS Computer").click
    sleep 0.5
    board = driver.find_elements(:class_name, "XCUIElementTypeImage")
    # 0 1 2
    # 3 4 5
    # 6 7 8

    blank_cells = [0, 1, 2, 3, 4, 5, 6, 7, 8]
    while blank_cells.count > 0 && driver.find_element(:class_name, "XCUIElementTypeStaticText")["value"].include?("Game is playing")
      sleep 0.5 # wait for computer's turn to finish
      if blank_cells.size < 9 # if all blank = 9, the first move, don't need check
        save_board_screenshot
        smart_crop_board_into_9_images(blank_cells)

        blank_cells.each do |x|
          row = x / 3
          col = x % 3
          img = ChunkyPNG::Image.from_file("/tmp/cell_#{row}_#{col}.png")
          blank_cells.delete(x) unless is_blank_cell(img)
        end
      end

      puts blank_cells.inspect
      empty_cell = blank_cells.sample
      board[empty_cell].click
      blank_cells.delete(empty_cell)
    end

    expect(driver.find_element(:class_name, "XCUIElementTypeStaticText")["value"]).to include("Game has ended!")
  end
end