Case Study: Image Manipulation in Automated End-to-End Tests

Automate an iOS Tic Tac Toe app with Appium 2 (XCUITest) Part 2: Player vs Computer.

Courtney Zhan
7 min readFeb 25, 2023

In a previous article, I walked through using Appium 2 to play a TicTacToe game (on iOS). That was Player vs Player mode, i.e. we can plan the moves to get desired output.

I will add a more dynamic twist in this article by choosing Player vs Computer mode. Please check the previous article’s set-up guide if you want to follow along.

Test Case:

To complete a Tic Tac Toe game (on iOS) in the Player vs Computer mode using automated end-to-end test scripts.

This app has 9 cells, each can be one of three states:

  • blank
  • ✕ image
  • ⭕ image

The challenge of this test case is to tap an available (i.e. blank) cell, which are non-deterministic.

In Appium, to my knowledge, there is no way to retrieve the image’s name. Therefore, we can’t detect what is in the cell. This translates to an image detection problem.

The test script for this article is in Ruby, the best language for scripting automated tests. There are intuitive image manipulation Ruby libraries (called gems), which I will use to accomplish the task.

Test Design

1. Take a screenshot of the app after each move, and save it to a file.

2. Crop the board (square shape) out of the screenshot.

3. Crop the board to generate nine equal-sized images (to represent the cells).

4. Analyze each cell image to determine whether it is blank or not.
The easiest way to do it is via colour detection. Please note, whilst it appears just red or blue to human eyes, there are many more colours (anti-aliasing). We can use the number of unique colours to differentiate blank and played cells.

5. On the player’s turn, place a token in a random blank cell until the game ends.

Preparation

Install required gems.

1. ImageScience

This is an image cropping library.

% brew install FreeImage

% gem install — no-document image_science

2. ChunkyPNG

This gem gets PNG info, including each cell’s colours.

% gem install-no-document chunky_png

Steps

  1. Save a screenshot of the app in Appium.

Appium has a built-in function screenshot-as which saves a screenshot of the iOS screen.

# screenshot then save as PNG in /tmp/base64.png
png_base64 = driver.screenshot_as(:base64)
File.open("/tmp/base64.png", "wb").puts(Base64.decode64(png_base64))

The screenshot is encoded as Base64 format. To save it as a PNG file, decode the Base64 and save it in binary file format (with wb).

Example base64.png screenshot

2. Crop the board (square shape) out of the screenshot.

Knowing the exact area of where to crop the board is finicky. I had to manually open the saved image and figure out where precisely the board began and ended, as well as the board’s dimensions.

The board was 1074 x 1074 pixels. The board offsets were 48 pixels (from the left) and 688 pixels (from the top). With this, we can crop the board.

width = 1074
height = 1074
# cut images with ImageScience library
ImageScience.with_image("/tmp/base64.png") do |img|
img.with_crop(48, 688, 48 + width, 688 + height) do |crop|
# save the board to /tmp/board.png
crop.save "/tmp/board.png"
end
end
Example board.png

3. Crop the board to generate nine equal-sized images.

In this step, we take the previously generated board.png and slice it into 9 equal squares.

cell_width = width / 3
cell_height = height / 3

ImageScience.with_image("/tmp/board.png") do |img|
cell_list.each do |x|
row = x / 3
col = x % 3
img.with_crop(col * cell_width, row * cell_height, (col + 1) * cell_width, (row + 1) * cell_height) do |crop|
# save each cell and name the file with the row-column coordinates
crop.save "/tmp/cell_#{row}_#{col}.png"
end
end
end
Example cell_0_0.png (upper-left-most cell)

4. Analyze each cell image to determine whether it is blank or not?

To check if a cell is blank, I used ChunkyPNG to returns the number of unique colours present. If the number is low (<1000), then we can assume the cell is blank.

Because of the red cell border and anti-aliasing, there are around 200 colours for a blank cell. I’ve set the limit to 1000 just to be safe.

def is_blank_cell(img)
height = img.dimension.height
width = img.dimension.width
puts "height: #{height}, width: #{width}"
color_codes = []
height.times do |i|
width.times do |j|
arr = [ChunkyPNG::Color.r(img[j, i]), ChunkyPNG::Color.g(img[j, i]), ChunkyPNG::Color.b(img[j, i])]
color_codes << "\##{arr.map { |x| x.to_s(16).rjust(2, "0") }.join.upcase}"
end
end
color_codes.uniq!
return color_codes.size < 1000
end

5. On the player’s turn, place a token in a random blank cell until the game is ended.

As shown in the previous article, playing a cell is as simple as tapping it. The problem in this step is choosing a random blank cell.

I decided to record all the blank cells in an array named blank_cells. I used the sample function to randomly select one blank cell, as shown below. A bit of programming required here.

blank_cells = [0, 1, 2, 3, 4, 5, 6, 7, 8]

# save and crop the board
blank_cells.each do |x|
row = x / 3 # convert index to coordinates
col = x % 3
img = ChunkyPNG::Image.from_file("/tmp/cell_#{row}_#{col}.png")
# remove all cells that are not blank
blank_cells.delete(x) unless is_blank_cell(img)
end

# randomly select one of the blank cells and play it
empty_cell = blank_cells.sample
board[empty_cell].click

The above code block only does one turn. To play a full game, we should add a loop so that the script keeps going until the game finishes.

6. Determine whether a game is completed or not.

The app has a game status text, it’s two states are “Game is playing” or “Game has ended!”

Game in progress text (on Computer’s turn)
Game completed text

We can loop the player’s turn based on the value of the game status text.

blank_cells = [0, 1, 2, 3, 4, 5, 6, 7, 8]

while blank_cells.count > 0 && driver.find_element(:class_name, "XCUIElementTypeStaticText")["value"].include?("Game is playing")
sleep 0.5 # wait for computer's turn to finish
if blank_cells.size < 9 # if all blank = 9, the first move, don't need check
save_board_screenshot
smart_crop_board_into_9_images(blank_cells)

blank_cells.each do |x|
row = x / 3
col = x % 3
img = ChunkyPNG::Image.from_file("/tmp/cell_#{row}_#{col}.png")
blank_cells.delete(x) unless is_blank_cell(img)
end
end

puts blank_cells.inspect
empty_cell = blank_cells.sample
board[empty_cell].click
blank_cells.delete(empty_cell)
end

expect(driver.find_element(:class_name, "XCUIElementTypeStaticText")["value"]).to include("Game has ended!")

Video Execution

Animated GIF

YouTube video.

Complete Test Script

load File.dirname(__FILE__) + "/../test_helper.rb"
require "base64"
require "chunky_png"
# brew install FreeImage, then gem install image_science, https://github.com/seattlerb/image_science
require "image_science"

describe "Player Vs Computer Games" do
include TestHelper

before(:all) do
@driver = Appium::Driver.new(appium_opts).start_driver
end

after(:all) do
driver.quit unless debugging?
end

def smart_crop_board_into_9_images(cell_list)
width = 1074
height = 1074
# cut images with ImageScience library
ImageScience.with_image("/tmp/base64.png") do |img|
img.with_crop(48, 688, 48 + width, 688 + height) do |crop|
crop.save "/tmp/board.png"
end
end

cell_width = width / 3
cell_height = height / 3

ImageScience.with_image("/tmp/board.png") do |img|
cell_list.each do |x|
row = x / 3
col = x % 3
img.with_crop(col * cell_width, row * cell_height, (col + 1) * cell_width, (row + 1) * cell_height) do |crop|
crop.save "/tmp/cell_#{row}_#{col}.png"
end
end
end
end

def save_board_screenshot
png_base64 = driver.screenshot_as(:base64)
File.open("/tmp/base64.png", "wb").puts(Base64.decode64(png_base64))
end

def is_blank_cell(img)
height = img.dimension.height
width = img.dimension.width
puts "height: #{height}, width: #{width}"
color_codes = []
height.times do |i|
width.times do |j|
arr = [ChunkyPNG::Color.r(img[j, i]), ChunkyPNG::Color.g(img[j, i]), ChunkyPNG::Color.b(img[j, i])]
color_codes << "\##{arr.map { |x| x.to_s(16).rjust(2, "0") }.join.upcase}"
end
end
color_codes.uniq!
return color_codes.size < 1000
end

it "Player vs Computer, Taking screenshot" do
driver.find_element(:name, "Player VS Computer").click
sleep 0.5
board = driver.find_elements(:class_name, "XCUIElementTypeImage")
# 0 1 2
# 3 4 5
# 6 7 8

blank_cells = [0, 1, 2, 3, 4, 5, 6, 7, 8]
while blank_cells.count > 0 && driver.find_element(:class_name, "XCUIElementTypeStaticText")["value"].include?("Game is playing")
sleep 0.5 # wait for computer's turn to finish
if blank_cells.size < 9 # if all blank = 9, the first move, don't need check
save_board_screenshot
smart_crop_board_into_9_images(blank_cells)

blank_cells.each do |x|
row = x / 3
col = x % 3
img = ChunkyPNG::Image.from_file("/tmp/cell_#{row}_#{col}.png")
blank_cells.delete(x) unless is_blank_cell(img)
end
end

puts blank_cells.inspect
empty_cell = blank_cells.sample
board[empty_cell].click
blank_cells.delete(empty_cell)
end

expect(driver.find_element(:class_name, "XCUIElementTypeStaticText")["value"]).to include("Game has ended!")
end
end

--

--