Automatic Testing of GAE Applications

(This post is part of the series Web Application on Resources in the Cloud.)

I am not a purist when it is time to apply a development method, I am a practical guy!

Update 2009/11/20
At one point, I switched to App Engine Java (see my post Google App Engine Meets Java). One of my argument at that time was the lack of tools Python-side to produce code coverage numbers... I've just posted a lengthy article about Unit tests, Mock objects, and App Engine for the Java back-end. The techniques and the code I share in this post allow me to reach the mystical 100% of code coverage, for a project with over 10,000 lines for the business code. See you there ;)

For example, I know by experience how important testing code as soon it is produced. And these tests must run after each code delivery to detect defects and regressions as soon as possible [1]. I allocate as much time to write tests (unit tests most of the time, functional tests occasionally) as I allocate to write code. And no task is considered complete if the corresponding tests do not cover 100%* of its code! I admire adopters of the Test Driven Development (TDD [2]) approach.

When searching information to test Google App Engine (GAE) applications, I was happy to find a post [3] written by Josh Cronemeyer who says:

My favorite way to start any project is by doing TDD. 

If you follow GAE tutorial [4], you can create a correctly designed application implementing the Model-View-Controller (MVC) pattern [5]. Testing such an application which organizes its behavior in different logical area means testing each areas independently:

1. Testing Python models
2. Testing Python business logic
3. Testing data injection in Django templates
4. Testing rendered HTML pages
5. Verifying the use-cases' implementation

In the following sections, I am going to summarize information I gathered. If I missed any element, please, let me know by posting a comment ;)

Testing Python models

Verifying that your models work as expected is really important. If we can rely on the robustness of Google Big Database implementation (after all, more and more of their applications run on App Engine), you need at least to detect regressions that our future refactoring operations might introduce.

To learn how to setup your environment, read the detailed post of Josh Cronemeyer: Unit Test Your Google App Engine Models.

What follows is some information to get you started writing unit tests against a GAE model. First, a list of the tools you need to install.

• Nose is a tool for running your python unit tests.
• NoseGAE is a plugin for nose that bootstraps the GAE environment.
  

An alternative is to use gaeunit [6] but tests might impact the persistence layer.

Josh Cronemeyer suggests to create a new stub data store before running all tests (function called in your test setUp()):

from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore_file_stub

def clear_datastore(self):
   # Use a fresh stub datastore.
   apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
   stub = datastore_file_stub.DatastoreFileStub('appid', '/dev/null', '/dev/null')
   apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', stub)

Testing Python business logic

What do I mean by Python business logic? This is the piece of code that deals with the end-user requests and that does some computations before pinging the persistence layer. For example, a function verifying the format of e-mail addresses (using regular expressions) belongs to that category.

This last 6 years, I have mainly used Java as the programming language to develop the server-side logic. Java is a simple and powerful programming language. Java benefits from big companies' support and has a large set of helper tools. As helpers, I can count: ant and maven, cruisecontrol and hudson, junit and code coverage, tools for static and dynamic code reviews, powerful IDEs, etc.

To developers starting to write unit tests with the target of covering 100% of the code, I have always suggested to read at least the chapter 7 of the book “JUnit in Action” [7], written by Vincent Massol. He introduces the Inversion Of Control (IOC) pattern and the Mock object concept:

Mock objects (or mocks for short) are perfectly suited for testing a portion of code logic in isolation from the rest of the code. Mocks replace the objects with which your methods under test collaborate, thus offering a layer of isolation. 

Python has also many frameworks to mock objects. Among them, Python Mocker seems to be very popular: it consists in recording behaviors and setting expectations on mock object, before letting them being used by real functions. See [9] for a detailed “how to use Mocker.”

Reminder: with the tip from Cronemeyer (see above with the stub for the data store, for more details look at [10]), there is no need to mock the data store.

Unit Test Sample in Python

The following piece of code defines utility methods which return 1) a list of supported languages and 2) a dictionary with localized labels (fallback on English one).

# -*- coding: utf-8 -*-

import en
import fr

def getLanguages():
    return {
        "en": en._getDictionary()["_language"],
        "fr": fr._getDictionary()["_language"],
    }

def getDictionary(locale):
    global dict
    if locale == "fr":
        dict = fr._getDictionary()
    else:
        dict = en._getDictionary()
    return dict

The series of tests in the following pieces of code verifies the list of languages contains at least English and French, verifies that the expected dictionaries contains the mandatory key with their name (“English” and “Français”), and that requiring an unexpected dictionary gives the English one.

# -*- coding: utf-8 -*-

import unittest
from prodcons.i18n import accessor

class SuccessFailError(unittest.TestCase):

    def test_getLanguages_I(self):
        """Verify at least 2 languages {en, fr} are in the dictionary"""
        self.assertTrue(len(accessor.getLanguages()) >= 2)
        self.assertEqual("English", accessor.getLanguages()["en"])
        self.assertEqual("Français", accessor.getLanguages()["fr"])

    def test_getDictionary_I(self):
        """Verify we can get a valid {en} dictionary"""
        self.assertTrue(accessor.getDictionary("en"))
        self.assertEqual("English", accessor.getDictionary("en")["_language"]) # Mandatory key

    def test_getDictionary_II(self):
        """Verify we can get a valid {fr} dictionary"""
        self.assertTrue(accessor.getDictionary("fr"))
        self.assertEqual("Français", accessor.getDictionary("fr")["_language"]) # Mandatory key

    def test_getDictionary_III(self):
        """Verify the fallback on the English dictionary"""
        self.assertTrue(accessor.getDictionary("no_NO"))
        self.assertEqual(accessor.getDictionary("en"), accessor.getDictionary("no_NO"))

Testing data injection in Django templates

Django comes with its own test runner! So its templates can be simulated in a stand alone mode.

With the help of Mocker [8], as described in [10], it is relatively easy to verify that your templates extract data as expected. If there is an unexpected data access to the Mock object, or if one attribute has been forgotten, the mock object will report it (a call to verify() ensure that all expectations were met.)

Testing rendered HTML pages

To verify that Django templates display the data as expected. Selenium [11] offers probably the best framework (and it's free):

• Selenium IDE which runs as a Firefox extension and that can record, edit, and debug test scripts.
• Selenium Remote Control (RC): it is a server that starts/stops browsers and that makes them running test scripts.
• Selenium Grid: to control and run tests on remote Selenium RC instances (on WinXP, Win7, RedHat, Suse, etc.)
  

At that step, Selenium tests are considered being functional tests because they involve many parts of the application (do not work in isolation). It's fine to run them extensively in development environment, but should be carefully used in production (see next section).

The following presentation (3:30) shows how I write a test against my local deployment. The basic test ensure the language switcher works correctly:

• Starting from the English homepage, it checks page title (in head>title and in div#title) against the label “Producer-Consumer”.
• After having switched to the French page, it verifies that the URL has been updated correctly (no more lang=en, but lang=fr in place). It checks also the page title against “Producteur-Consommateur”.
• After having switched back to the English page, it verifies the URL has been updated correctly.
  

Rapid definition of a test case with Selenium IDE.


Check points defined with Selenium IDE.

Verifying the use-cases' implementation

In Agile development, a sprint is a period of time allocated to complete development tasks. Each task or group of tasks implement a use-case, a story. Use-cases are used to validate the deliveries. Here is a simple story:

End-users fine-tune searches with the prefixes name:, producer:, and unit-price: 

At the beginning of a project, use cases are simple and can be covered by automatic tests. After a while, use cases become more complex and running them takes quite a long time and requires a heavy setup. These complex use cases are usually processed manually by human operators (Quality Engineers).

If these qualified workers can focus on complex use cases (all dumb ones are processed automatically), they have more possibility to find real bugs, I mean behaviors that coders forgot to cover, that product owners forgot to specify, etc. Sooner these bugs are discovered lower is their cost in terms of engineer time spent to fix them and in term of lost credibility!

I hope this helps!

Update 2009/01/30:
Josh, alias Shlomo, updated his post with an excellent reference of a discussion thread in Google App Engine group: Testing Recommendations. Look specifically at two messages posted by Andy about using Selenium tools, like this one:

Selenium-RC is not necessary since Selenium core can be added to your App Engine project directory and run from there. Collected steps below to save you time:

1. Download the core: http://selenium-core.openqa.org/download.jsp
2. Unzip selenium core zip file
3. Copy this selenium core directory into your App path somewhere.
4. Edit your app.yaml file to permit static access (static because it doesn't need a sever interpreter. selenium is written in javascript, so it runs in your browser.) Under "handlers:", add something like this:
   - url: /selenium
  static_dir: ./tests/SeleniumTests/selenium-core-0.8.3

static_dir is where I copied my selenium-core directory. To access selenium, I now open the app engine url, http://localhost:8080/selenium/index.html and run the TestRunner. From there, open your test suite and go. I had trouble loading individual tests for some reason, but suites work.

I could have of course done all of this using the Firefox Selenium-IDE plugin with fewer steps, but using the core approach does make it cross browser supporting.

I'm still trying to figure out how to use Python for this. I'll continue that in a separate message. 

A+, Dom
--
Note:

• Providing 100% of code coverage by unit or functional tests is sometimes not possible. The test case shutting down remotely the application server to verify that pending transactions are persisted correctly cannot, for example, be scripted for automation. In such a situation, we should rely on manual testing. The goal of test automation is to provide the best coverage possible, so manual testers can focus on edge cases and on trying to discover unexpected issues.
  

--
Sources

1. Agile: SCRUM is hype, but XP is as important... (another post to be published soon)
   
2. Test Driven Development approach description on Wikipedia, and information on the book Test Driven Development by Kent Beck
   
3. Unit Test Your Google App Engine Models by Josh Cronemeyer, and Josh's short bio from a speech he gave at OSCON in 2007.
4. Google App Engine tutorial on Google Code.
   
5. MVC Pattern applied to GAE Applications... (another post to be published soon)
6. gaeunit is a test environment that runs on development environment (not on Google infrastructure).
   
7. JUnit in Action, by Vincent Massol, edited by Manning Publications Co. Chapter 7, freely available, explains the IOC pattern and the benefits of using Mock objects. Vincent Massol is now technical lead of the open source project XWiki.
8. Python Mocker: Mock object framework which allows to record behaviors and expectations before replaying them with real functions.
   
9. Unit tests for Google App Engine apps, by App Engine Fan.
10. Proper Unit Testing of App Engine/Django, by App Engine Guy
11. The open-source framework for Web application automatic testing: Selenium IDE, Selenium RC, and Selenium Grid. Check the core features for information on the automating the tests -- See also a Japanese wiki on Selenium: JavaScud / Selenium IDE.