For my [still a secret] project which is running on Google App Engine infrastructure [1], I want to make it as solid as possible from the beginning by applying most of the best practices of the Agile methodology [2].
With the release of the App Engine Java SDK 1.2.8 (read release notes, I had to update my code and this post on two points:
- Without the specification of the JDO inheritance type, the environment assumes it's
superclass-table. This type is not supported by App Engine. Onlysubclass-tableandcomplete-tableare supported. In theEntityclass described below, I had to add@Inheritance(strategy = InheritanceStrategy.SUBCLASS_TABLE). Read the documentation about Defining data classes for more information. - With the automation of the task execution, the
MockAppEngineEnvironmentclass listed below had to be updated to serve an expected value when theQueueruns in the live environment. Read the details on the thread announcing the 1.2.8. SDK prerelease on Google Groups.
As written on my post from September 18, I had to develop many mock classes to keep reaching the mystical 100% of code coverage (by unit tests) [3]. A good introduction of mock objects is given by Vincent Massol in his book “JUnit in Action” [4]. To summarize, mock objects are especially useful to inject behavior and force the code using them to exercise complex control flows.
Developing applications for Google App Engine is not that complex because the system has a good documentation and an Eclipse plug-in ease the first steps.
Use case description
Let's consider a simple class organization implementing a common J2EE pattern:
- A DTO class for a
Consumer; - The DAO class getting the
Consumerfrom the persistence layer, and sending it back with updates; and - A Controller class routing REST requests. The Controller is an element of the implemented MVC pattern
The code for the DTO class is instrumented with JDO annotations [5]:
Consumer DTO class
definition@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
@Inheritance(strategy = InheritanceStrategy.SUBCLASS_TABLE)
public class Consumer extends Entity {
@Persistent
private String address;
@Persistent
private String displayName;
@Persistent
private String email;
@Persistent
private String facebookId;
@Persistent
private String jabberId;
@Persistent
private Long locationKey;
@Persistent
private String twitterId;
/** Default constructor */
public Consumer() {
super();
}
/**
* Creates a consumer
* @param in HTTP request parameters
*/
public Consumer(JsonObject parameters) {
this();
fromJson(parameters);
}
public String getAddress() {
return address;
}
public void setAddress(String address) {
this.address = address;
}
//...
}
My approach for the DAO class is modular:
- When the calling code is doing just one call, like the
ConsumerOperations.delete(String)method deleting the identifiedConsumerinstance, the call can be done without the persistence layer knowledge. - When many calls to the persistence layer are required, the DAO
API offers the caller to pass a
PersistenceManagerinstance that can be re-used from call to call. With the combination of thedetachable="true"parameter specified in the JDO annotation for theConsumerclass, it saves many cycles.
ConsumerOperations
DAO class definition/**
* Persist the given (probably updated) resource
* @param consumer Resource to update
* @return Updated resource
* @see ConsumerOperations#updateConsumer(PersistenceManager, Consumer)
*/
public Consumer updateConsumer(Consumer consumer) {
PersistenceManager pm = getPersistenceManager();
try {
// Persist updated consumer
return updateConsumer(pm, consumer);
}
finally {
pm.close();
}
}
/**
* Persist the given (probably updated) resource while leaving the given persistence manager open for future updates
* @param pm Persistence manager instance to use - let opened at the end to allow possible object updates later
* @param consumer Resource to update
* @return Updated resource
*/
public Consumer updateConsumer(PersistenceManager pm, Consumer consumer) {
return pm.makePersistent(consumer);
}
The following piece of the abstract class BaseOperations
shows the accessor made availabe to any controller code to get one
handle of a valid PersistenceManager instance.
BaseOperations
DAO class definition/**
* Accessor isolated to facilitate tests by IOP
* @return Persistence manager instance
*/
public PersistenceManager getPersistenceManager() {
PersistenceManager pm = getPersistenceManagerFactory().getPersistenceManager();
pm.setDetachAllOnCommit(true);
pm.setCopyOnAttach(false);
return pm;
}
To finish the use case setup, here is a part of the controller code which deals with incoming HTTP requests and serves or operates accordingly. This specific piece of code replies to a GET request like:
- Invocation:
http://<host:port>/API/Consumer/43544" - Response:
{key:43544, displayName:"John", address:"75, Queen, Montréal, Qc, Canada", locationKey:3245, location: {id:3245, postalCode:"H3C2N6", countryCode:"CA", latitude:43.3, longitude:-73.4}, ...}
ConsumerRestlet Controller class definition@Override
protected JsonObject getResource(JsonObject parameters, String resourceId, User loggedUser) throws DataSourceException {
PersistenceManager pm = getBaseOperations().getPersistenceManager();
try {
// Get the consumer instance
Consumer consumer = getConsumerOperations().getConsumer(pm, Long.valueOf(resourceId));
JsonObject output = consumer.toJson();
// Get the related information
Long locationKey = consumer.getLocationKey();
if (locationKey != null) {
Location location = getLocationOperations().getLocation(pm, locationKey);
output.put(Consumer.LOCATION, location.toJson());
}
// Return the complete set of information
return output;
}
finally {
pm.close();
}
}
Simple mock
Now, it's time to test! To start slowly, let's deal with the Restlet getResource() method to verify:
- Just one and only one instance of
PersistenceManageris loaded by the function; - The
PersistenceManagerinstance is cleanly closed at the end of the process; - There's a call issued to get the identified
Consumerinstance; - There's possibly a call issued to get the identified
Locationinstance; - The output value has the expected information.
In the corresponding unit test series, we don't want to interfere with the App Engine infrastructure (the following chapter will address that aspect). So we'll rely on a mock for the PersistenceManager class that will be injected into the ConsumerRestlet code. The full source of this class is available on my open source project two-tiers-utils: javax.jdo.MockPersistenceManager.
PersistenceManager classpublic class MockPersistenceManager implements PersistenceManager {
private boolean closed = false; // To keep track of the "closed" state
public void close() {
closed = true;
}
public boolean isClosed() {
return closed;
}
// ...
}
Here are the unit tests verifying the different flow paths:
- When an exception is thrown, because the back-end does not serve the data for example;
- When the
Consumerinstance returns without location coordinates; - When the
Consumerinstance is fully documented.
ConsumerRestlet.getResource() method@Test(expected=IllegalArgumentException.class)
public void testUnexpectedError() {
// Test prepration
final PersistenceManager pm = new MockPersistenceManager();
final BaseOperations baseOps = new BaseOperations() {
boolean askedOnce = false;
@Override
PersistenceManager getPersistenceManager() {
if (askedOnce) {
fail("Expects only one call");
}
askedOnce = true;
return pm;
}
};
final Long consumerId = 12345L;
final ConsumerOperations consumerOps = new ConsumerOperations() {
@Override
Consumer getConsumer(PersistenceManager pm, Long id) {
assertEquals(consumerId, id);
throw new IllegalArgumentException("Done in purpose!");
}
};
ConsumerRestlet restlet = new ConsumerRestlet() {
@Override BaseOperation getBaseOperations() { return baseOps; }
@Override ConsumerOperation getConsumerOperations() { return consumerOps; }
}
// Test itself
JsonObject response = restlet.getResource(null, consumerId.toString, null);
}
@Test
public void testGettingOneConsumer() {
// Test prepration
final PersistenceManager pm = new MockPersistenceManager();
final BaseOperations baseOps = new BaseOperations() {
boolean askedOnce = false;
@Override
PersistenceManager getPersistenceManager() {
if (askedOnce) {
fail("Expects only one call");
}
askedOnce = true;
return pm;
}
};
final Long consumerId = 12345L;
final ConsumerOperations consumerOps = new ConsumerOperations() {
@Override
Consumer getConsumer(PersistenceManager pm, Long id) {
assertEquals(consumerId, id);
Consumer consumer = new Consumer();
consumer.setId(consumerId);
return consumer;
}
};
final Long locationId = 67890L;
final LocationOperations locationOps = new LocationOperations() {
@Override
Location getLocation(PersistenceManager pm, Long id) {
fail("Call not expected here!");
return null;
}
};
ConsumerRestlet restlet = new ConsumerRestlet() {
@Override BaseOperation getBaseOperations() { return baseOps; }
@Override ConsumerOperation getConsumerOperations() { return consumerOps; }
@Override LocationOperation getLocationOperations() { return locationOps; }
}
// Test itself
JsonObject response = restlet.getResource(null, consumerId.toString, null);
// Post-test verifications
assertTrue(pm.isClosed());
assertNotSame(0, response.size());
assertTrue(response.containsKey(Consumer.ID);
assertEquals(consumerId, response.getLong(Consumer.ID));
}
@Test
public void testGettingConsumerWithLocation() {
// Test prepration
final PersistenceManager pm = new MockPersistenceManager();
final BaseOperations baseOps = new BaseOperations() {
boolean askedOnce = false;
@Override
PersistenceManager getPersistenceManager() {
if (askedOnce) {
fail("Expects only one call");
}
askedOnce = true;
return pm;
}
};
final Long consumerId = 12345L;
final Long locationId = 67890L;
final ConsumerOperations consumerOps = new ConsumerOperations() {
@Override
Consumer getConsumer(PersistenceManager pm, Long id) {
assertEquals(consumerId, id);
Consumer consumer = new Consumer();
consumer.setId(consumerId);
consumer.setLocationId(locationId);
return consumer;
}
};
final LocationOperations locationOps = new LocationOperations() {
@Override
Location getLocation(PersistenceManager pm, Long id) {
assertEquals(locationId, id);
Location location = new Location();
location.setId(locationId);
return location;
}
};
ConsumerRestlet restlet = new ConsumerRestlet() {
@Override BaseOperation getBaseOperations() { return baseOps; }
@Override ConsumerOperation getConsumerOperations() { return consumerOps; }
@Override LocationOperation getLocationOperations() { return locationOps; }
}
// Test itself
JsonObject response = restlet.getResource(null, consumerId.toString, null);
// Post-test verifications
assertTrue(pm.isClosed());
assertNotSame(0, response.size());
assertTrue(response.containsKey(Consumer.ID);
assertEquals(consumerId, response.getLong(Consumer.ID));
assertTrue(response.containsKey(Consumer.LOCATION_ID);
assertEquals(locationId, response.getLong(Consumer.LOCATION_ID));
assertTrue(response.containsKey(Consumer.LOCATION);
assertEquals(consumerId, response.getJsonObject(Consumer.LOCATION).getLong(Location.ID));
}
Note that I would have been able to override just the PersistenceManager class to have the Object getObjectById(Object arg0) method returning the expected exception, Consumer, and Location instances. But I would have pass over the strict limit of a unit test by then testing also the behavior of the ConsumerOperations.getConsumer() and LocationOperations.getLocation() methods.
App Engine environment mock
Now, testing the ConsumerOperations class offers a better challenge.
As suggested above, I could override many pieces of the PersistenceManager class to be sure to control the flow. But to do a nice simulation, I almost need to have the complete specification of the Google App Engine infrastructure to be sure I mock it correctly. This is especially crucial when processing Query because Google data store has many limitations [6] that others traditional database, like MySQL, don't have...
Because this documentation is partially available and because Google continues to update its infrastructure, I looked for a way to use the standalone environment made available with the App Engine SDK [1]. This has not been easy because I wanted to have the test running independently from the development server itself. I found first some documentation on Google Code website: Unit Testing With Local Service Implementations, but it was very low level and did not fit with my JDO instrumentation of the DTO classes. Hopefully, I found this article JDO and unit tests from App Engine Fan, a great community contributor I mentioned many times in previous posts!
By cooking information gathered on Google Code website and on App Engine Post, I've produced a com.google.apphosting.api.MockAppEngineEnvironment I can use for my JUnit4 tests.
ConsumerRestlet.getResource() methodpackage com.google.apphosting.api;
// import ...
/**
* Mock for the App Engine Java environment used by the JDO wrapper.
*
* These class has been build with information gathered on:
* - App Engine documentation: http://code.google.com/appengine/docs/java/howto/unittesting.html
* - App Engine Fan blog: http://blog.appenginefan.com/2009/05/jdo-and-unit-tests.html
*
* @author Dom Derrien
*/
public class MockAppEngineEnvironment {
private class ApiProxyEnvironment implements ApiProxy.Environment {
public String getAppId() {
return "test";
}
public String getVersionId() {
return "1.0";
}
public String getEmail() {
throw new UnsupportedOperationException();
}
public boolean isLoggedIn() {
throw new UnsupportedOperationException();
}
public boolean isAdmin() {
throw new UnsupportedOperationException();
}
public String getAuthDomain() {
throw new UnsupportedOperationException();
}
public String getRequestNamespace() {
return "";
}
public Map getAttributes() {
Map out = new HashMap();
// Only necessary for tasks that are added when there is no "live" request
// See: http://groups.google.com/group/google-appengine-java/msg/8f5872b05214...
out.put("com.google.appengine.server_url_key", "http://localhost:8080");
return out;
}
};
private final ApiProxy.Environment env;
private PersistenceManagerFactory pmf;
public MockAppEngineEnvironment() {
env = new ApiProxyEnvironment();
}
/**
* Setup the mock environment
*/
public void setUp() throws Exception {
// Setup the App Engine services
ApiProxy.setEnvironmentForCurrentThread(env);
ApiProxyLocalImpl proxy = new ApiProxyLocalImpl(new File(".")) {};
// Setup the App Engine data store
proxy.setProperty(LocalDatastoreService.NO_STORAGE_PROPERTY, Boolean.TRUE.toString());
ApiProxy.setDelegate(proxy);
}
/**
* Clean up the mock environment
*/
public void tearDown() throws Exception {
// Verify that there's no pending transaction (ie pm.close() has been called)
Transaction transaction = DatastoreServiceFactory.getDatastoreService().getCurrentTransaction(null);
boolean transactionPending = transaction != null;
if (transactionPending) {
transaction.rollback();
}
// Clean up the App Engine data store
ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
if (proxy != null) {
LocalDatastoreService datastoreService = (LocalDatastoreService) proxy.getService("datastore_v3");
datastoreService.clearProfiles();
}
// Clean up the App Engine services
ApiProxy.setDelegate(null);
ApiProxy.clearEnvironmentForCurrentThread();
// Report the issue with the transaction still open
if (transactionPending) {
throw new IllegalStateException("Found a transaction nor commited neither rolled-back." +
"Probably related to a missing PersistenceManager.close() call.");
}
}
/**
* Creates a PersistenceManagerFactory on the fly, with the exact same information
* stored in the /WEB-INF/META-INF/jdoconfig.xml file.
*/
public PersistenceManagerFactory getPersistenceManagerFactory() {
if (pmf == null) {
Properties newProperties = new Properties();
newProperties.put("javax.jdo.PersistenceManagerFactoryClass",
"org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory");
newProperties.put("javax.jdo.option.ConnectionURL", "appengine");
newProperties.put("javax.jdo.option.NontransactionalRead", "true");
newProperties.put("javax.jdo.option.NontransactionalWrite", "true");
newProperties.put("javax.jdo.option.RetainValues", "true");
newProperties.put("datanucleus.appengine.autoCreateDatastoreTxns", "true");
newProperties.put("datanucleus.appengine.autoCreateDatastoreTxns", "true");
pmf = JDOHelper.getPersistenceManagerFactory(newProperties);
}
return pmf;
}
/**
* Gets an instance of the PersistenceManager class
*/
public PersistenceManager getPersistenceManager() {
return getPersistenceManagerFactory().getPersistenceManager();
}
}
With such a class, the unit test part is easy and I can build complex test cases without worrying about the pertinence of my mock classes! That's really great.
TestConsumerOperations classpublic class TestConsumerOperations {
private MockAppEngineEnvironment mockAppEngineEnvironment;
@Before
public void setUp() throws Exception {
mockAppEngineEnvironment = new MockAppEngineEnvironment();
mockAppEngineEnvironment.setUp();
}
@After
public void tearDown() throws Exception {
mockAppEngineEnvironment.tearDown();
}
@Test
public void testCreateVI() throws DataSourceException, UnsupportedEncodingException {
final String email = "unit@test.net";
final String name = "Mr Unit Test";
Consumer newConsumer = new Consumer();
newConsumer.setDisplayName(name);
newConsumer.setEmail(email);
assertNull(newConsumer.getId());
// Verify there's no instance
Query query = new Query(Consumer.class.getSimpleName());
assertEquals(0, DatastoreServiceFactory.getDatastoreService().prepare(query).countEntities());
// Create the user once
ConsumerOperations ops = new ConsumerOperations();
Consumer createdConsumer = ops.createConsumer(newConsumer);
// Verify there's one instance
query = new Query(Consumer.class.getSimpleName());
assertEquals(1, DatastoreServiceFactory.getDatastoreService().prepare(query).countEntities());
assertNotNull(createdConsumer.getId());
assertEquals(email, createdConsumer.getEmail());
assertEquals(name, createdConsumer.getName());
}
// ...
}
Conclusion
As a big fan of TDD, I'm now all set to cover the code of my [still a secret] project efficiently. It does not mean everything is correct, more that everything I thought about is correctly covered. At the time of this writing, just for the server-side logic, the code I produced covers more than 10,000 lines and the unit tests bring an additional set of 23,400 lines.
When it's time to refactor a bit or to add new features (plenty of them are aligned in my task list ;), I feel comfortable because I know I can detect most of regressions (if not all) after 3 minutes of running the test suite.
If you want to follow this example, feel free to get the various mock classes I have added to my two-tiers-utils open-source project. In addition to mock classes for the App Engine environment, you'll find:
- Basic mock classes for the servlet (see
javax.servlet) andjavamocks.iopackages -- I had to adopt the rootjavamocksbecause the JVM class loader does not accept the creation on the fly of classes in thejavaroot). - A mock class for
twitter4j.TwitterUser-- I needed a class with public constructor and a easy way to create a default account. - A series of mock class for David Yu's Project which I use to allow users with OpenID credentials to log in. Read the discussion I had with David on ways to test his code, in fact the code he produced and I customized for my own needs and for other security reasons.
For other details on my library, read my post Internationalization and my two-tiers-utils library.
I hope this helps.
A+, Dom
--
References:
- Google App Engine: the homepage and the SDK page.
- See my post on Agile: SCRUM is Hype, but XP is More Important... where I mentionned the following techniques: Continuous Integration (CI), Unit testing and code coverage (CQC), and Continuous refactoring.
- I know that keeping 100% as the target for code coverage numbers is a bit extreme. I read this article Don't be fooled by the coverage report soon after I started using Cobertura. In addition to reducing the exposition to bugs, the 100% coverage gives a very high chance to detect regressions before pushing the updates to the source control system!
- Vincent Massol; JUnit in Action; Editions Manning; www.manning.com/massol and Petar Tahchiev, Felipe Leme, Vincent Massol, and Gary Gregory; JUnit in Action, Second Edition; Editions Massol; www.manning.com/tahchiev. I was used to asking any new developer joigning my team to read at least this chapter 7: Testing in isolation with mock objects.
- JDO stands for Java
Data Objects and is an attempt abstract the data storage manipulation.
The code is instrumented with Java annotations like
@Persistent, it is instrumented at compile time, and dynamically connect to the data source thanks few properties files. Look at the App Engine - Using the Datastore with JDO documentation for more information. - For general limitations, check this page Will it play in App Engine. For JDO related limitations, check the bottom of the page Usin JDO with App Engine.
A series of localized entries like 
To start investigations on mobile application development for Compuware, I have just acquired a first Android Dev Phone, also known as G1 [1]. Last week at Google I/O, Vic Gundotra delivered an Oprah event by offering to the audience a second generation Android phone, also known as G2 or HTC Magic [2, 3]. The G1 is more limited but it is still the only Android platform legally available. 
Une fois acquise Steltor par Oracle, j'ai travaillé pour une équipe californienne. Mon expérience des applications Web a retenue d'
C'est alors qu'IBM Rational m'a offert un poste d'architecte technique. Le projet n'était pas sensationnel (il est même mort depuis), mais les gens rencontrés et l'expérience de Big Blue ont été sans pareil. Chapeau à 
Présentement, je suis chez Compuware dans un groupe d'experts multi-disciplinaires et j'aide à définir les stratégies de l'entreprise dans les domaines des applications Web et des mobiles. Évoluant dans une compagnie très conservatrice, certaines situations ont tendance à me frustrer royalement, mais les résultats que j'arrive à obtenir par ma persévérance font que finalement l'équilibre est positif. Je ne sais pas combien de temps cela va durer, mais j'en apprécie la valeur. Et je remercie 
Du côté extra-professionnel, il y a eu mon passage dans le groupe RÉSULTATS, merci 
Je n'oublie pas non plus ma relativement récente entrée dans le monde du karaté au Dojo de Don Lorenzetti, pour le style Chito Ryu. C'est d'autant plus passionnant que toute la famille Derrien y participe : Sophie, Erwan, Goulven et moi-même chacun à des horaires différentes deux fois par semaine.
