Continuous GUI Integration Testing

I started reading "Growing Object-Oriented Software Guided by Tests" from Steve Freeman and Nat Pryce. The book is about effectively implementing Test-Driven-Development. One of the key statements in the book, is that you should always start with a (failing) end-to-end integration test also called a "walking skeleton" before you write any code. In part III the authors give a working example in form of a Java Swing application. They use a GUI test framework called Window Licker which can automatically test Swing applications. Nat Pryce, one of the authors, drives the Window Licker project.

Since I will start to develop Facebook games for Electronic Arts in Norway in a little while, I was looking into TDD and end-to-end testing in this area. Even though I will only be responsible for server backend development, a good continuously integrated working skeleton should probably test the GUI. There are several good GUI testing frameworks like Selenium or Watir. The tests can be written like standard unit tests and run automated. However, since they make use of API integration points, it is not trivial to test Flash based applications. Selenium for instance, uses the id-attributes to interact and validate HTML elements within a web page.

I came across a really interesting new framework called Project Sikuli. This library uses image recognition to identify screen elements. Sikuli is not really meant to only support GUI testing but rather to automate every-day tasks that involve a GUI. Everything that can be seen on the screen, can be scripted, not only the web browser. I can have a Sikuli script that clicks on "System > Administration > Nvidia X Server Settings" in Ubuntu and changes my screen resolution if I wanted to. Sikuli comes with a minimalistic IDE which allows you to take snapshot of a screen region that you want to interact with. In addition to that, using the IDE you can also test how well your snapshot region is recognized on the screen. The better it is recognized, the less often your unit tests will fail. Since the project is written in Java, there is also a Java library which enables you to write unit tests using Sikuli.

Let's give a working example. I want to write a integration test, that can be run automatically using Maven. The test should login to Facebook, go to the Farmville (actually a Flash game from our biggest competitor Zynga), click on the "Gifts" icon, click on "Sell" and finally click on the "Cancel" button. I am using Sikuli 0.10.2 in this example. First step to install all required dependencies in Ubuntu. Next step to download Sikuli 0.10.2 and add it to my local Maven repository since it is not available in any official Maven repository yet. Having Sikuli available as dependency, it is time for the first unit test.

package lantern.day.gui;

import edu.mit.csail.uid.FindFailed;
import edu.mit.csail.uid.Screen;
import org.junit.Before;
import org.junit.Test;

import java.net.URL;

/**
 * Verifies selling gifts can be cancelled in Farmville using
 * the Sikuli test framework.
 *
 * @author reiks, Jan 28, 2011
 */
public class FarmvilleCancelGiftSellingTest {

    private Screen _screen;

    @Before
    public void setUp() {
        _screen = new Screen();
    }

    @Test
    public void openFacebookHomepage() throws FindFailed {
        openChrome();
        goToURL("http://www.facebook.com/");
        login();
        startFarmville();
        cancelSellingOfGifts();
        closeChrome();
    }

    private String fullPath(final String fileName) {
        final URL url =  ClassLoader.getSystemResource(fileName);
        return url.getFile();
    }

    private void login() throws FindFailed {
        _screen.type(fullPath("email.png"), "password\n", 0);
    }

    private void openChrome() throws FindFailed {
        _screen.click(fullPath("chrome_icon.png"), 0);
        _screen.wait(fullPath("new_tab.png"));
    }

    private void closeChrome() throws FindFailed {
        _screen.click(fullPath("close.png"), 0);
    }

    private void goToURL(final String url) throws FindFailed {
        _screen.paste(fullPath("url_bar.png"), url);
        _screen.type(fullPath("url_bar.png"), "\n", 0);
    }

    private void startFarmville() throws FindFailed {
        _screen.click(fullPath("farmville_icon.png"), 0);
        _screen.wait(fullPath("gift.png"), 5000);
    }

    private void cancelSellingOfGifts() throws FindFailed {
        _screen.click(fullPath("gift.png"), 0);
        _screen.wait(fullPath("sell.png"), 1000);
        _screen.click(fullPath("sell.png"), 0);
        _screen.wait(fullPath("cancel.png"), 1000);
        _screen.click(fullPath("cancel.png"), 0);
    }
}

The FarmvilleCancelGiftSellingTest will open my local Google Chrome browser, go to http://www.facebook.com/, enter some test user credentials to log in, click the small Farmville icon to the left and so on. The test requires a couple of PNG files which Sikuli will use to identify the screen elements which it needs to click and validate. In this very simple test case, I am mainly using two methods of the edu.mit.csail.uid.Screen instance - click and wait. First I click an image then I wait for another image to be present on the screen. If the image I am waiting for is not found, the test will fail. I also use the type and paste functions in this class, to enter text into HTML text fields.

One important aspect of automated GUI integration testing, is the ability to check what went wrong if tests start to fail. Some developers have scripted their builds to take screen-shots during the test run, I decided to record the desktop as a video. An idea which I stole from the Window Licker website. Instead of using a JUnitResultFormatter, I wrote a RunListener that is being used from the maven-surefire-plugin. The code of the RunListener is almost identical to the ResultFormatter. Every test is being recorded using the Linux program recordmydesktop. If the unit test is failing, the video file is kept otherwise it is deleted. Here is the video for the FarmvilleCancelGiftSellingTest. It is running a bit fast as a video clip. During the actual test execution it is running much slower. *Update* thanks to EmbedPlus you can play the video in slow motion now. Just click the "Slow" button before you start playing the video. What you see in the video, is a Maven build starting a Chrome browser, executing and recording the test.




And finally the pom.xml I am using to run the Sikuli tests from Maven.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>lantern.day</groupId>
    <artifactId>gui-testing</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Automated GUI Testing using Project Sikuli</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.sikuli</groupId>
            <artifactId>sikuli-script</artifactId>
            <version>0.10.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.ant</groupId>
            <artifactId>ant-junit</artifactId>
            <version>1.8.2</version>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.4</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                    <showWarnings>false</showWarnings>
                    <showDeprecation>false</showDeprecation>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.7.2</version>
                <configuration>
                    <properties>
                        <property>
                            <name>listener</name>
                            <value>lantern.day.gui.junit.RecordingListener
    </value>
                        </property>
                    </properties>
                </configuration>
            </plugin>

        </plugins>
    </build>

</project>