MVC Pattern and REST API Applied to GAE Applications

(This post is part of the series Web Application on Resources in the Cloud.)

I have been developing applications for various categories of end-users since 1990. I coded front-ends in X/Motif [1] for Solaris, then in Borland OWL and Windows MFC for Windows [2], then HTML/JavaScript for Web browsers. Most of the time for good reasons, JavaScript has been considered as a hacking language: to do quick and dirty fixes. Anyway, I have always been able to implement the MVC pattern:

• My first enterprise Web applications was relying on a back-end written in C++ for a FastCGI run by Apache [3]. I wrote a HTML template language which, in some ways, was similar to the JSP concept [4].
• The second version of this administrative console was relying on JavaScript for the rendering (a-la Dojo parser) and was using frames to keep context and to handle asynchronous communication.
• Then I learned about XMLHttpRequest [5], and with a great team, I started building a JavaScript framework to be able to handle complex operations and screen organizations within an unique Web page. We came up with a large widget library and a custom set of data models.
• After a job change, I discovered Dojo [6] in its early days (0.2 to 0.4) and I closely followed the path to get to 1.0. With Dojo, I am now able to relax client-side because the widget library is really huge, while still easy to extend, and it has advanced data binding capabilities. Now, I can focus on the middle-tier to build efficient REST APIs.

Among the design patterns [7], Model-View-Controller (MVC) is maybe one of the most complex (it is a combination of many basic patterns: Strategy, Composite, Observer) and maybe the one with the most various interpretations. I did write my own guidelines when I ported the MVC pattern browser-side (proprietary work). Today, Kris Zip blog entry on SitePen side summarizes nicely my approach: Client/Server Model on the Web. The following diagrams illustrate the strategy evolution.

   
Credits: SitePen.

In Google App Engine documentation, Django [8] is the template language proposed to separate the View building from the Model manipulation. Django implements the “Traditional Web Application” approach illustrated above.

My initial concern about the traditional approach is the absence of a clean and public API to control the Model. It is not rare that APIs are considered as add-ons, as optional features. IMHO, APIs should be among the first defined elements: it helps defining the scope of the work, it helps defining iterations (in the Agile development spirit), it helps writing tests up-front, it helps isolating bottlenecks.

With the move of the MVC pattern browser-side, the need to define a server-side API becomes obvious. The Model is now ubiquitous: interactive objects client-side have no direct interaction with the back-end server, they just interact with the Model proxy. This proxy can fetch data on demand, can pre-fetch data, forwards most of the update requests immediately but can delay or abort (to be replayed later) idempotent ones.

My favorite API template for the server-side logic is the RESTful API [9]. It is simple to implement, simple to mock, and simple to test. For my side project [10], the repository contains descriptions of products made available by micro entrepreneurs (see table 1).

Table 1: Samples of RESTful HTTP requests

Verb	URL pattern	Description
GET	/API/products	Return HTTP code 200 and the list of all products, or the list of the ones matching the specified criteria.
GET	/API/products/{productId}	Return HTTP code 301 with the versioned URL for the identified product, or HTTP code 404 if the identified product is not found.
GET	/API/products/{productId}/{version}	Return HTTP code 200 and the attributes of the identified product for the specified version, or HTTP code 301 "Move Permanently" with a URL containing the new version information, or HTTP code 404 if the product is not found.
DELETE	/API/products/{productId}/{version}	Return HTTP code 200 if the deletion is successful, or HTTP code 303 or 404 if needed.
POST	/API/products	Return HTTP code 201 if the creation is successful with the new product identifier and its version number.
PUT	/API/products/{productId}/{version}	Return HTTP code 200 if the update is successful with the new product version number, or HTTP codes 303 or 404 if needed.

Table 2: Partial list of HTTP status codes (see details in [11]).

HTTP Code	Description
200	OK
201	Created
301	Moved Permanently
303	See Other “The response to the request can be found under another URI using a GET method. When received in response to a PUT or POST, it should be assumed that the server has received the data and the redirect should be issued with a separate GET message.”
304	Not Modified
307	Temporary Redirect
400	Bad Request
401	Unauthorized
404	Resource Not Found
410	Gone
500	Internal Server Error
501	Not Implemented

Parsing RESTful API for Google App Engine application is not difficult. It is just a matter of using regular expressions in the app.yaml configuration file and in the corresponding Python script file.

application: prod-cons
version: 1
runtime: python
api_version: 1

handlers:
- url: /API/products.*
  script: py/products.py

- url: /html
  static_dir: html

- url: /.*
  static_files: html/redirect.html
  upload: html/redirect.html

Code 1: Excerpt of the app.yaml configuration file.

# -*- coding: utf-8 -*-

import os

from google.appengine.api import users
from google.appengine.ext import db
from google.appengine.ext import webapp
from google.appengine.ext.webapp import template
from google.appengine.ext.webapp.util import run_wsgi_app

from prodcons import common
from prodcons import model

class ProductList(webapp.RequestHandler):
    def get(self):
        # Get all products
        # ...
    def post(self, productId, version):
        # Create new product
        # ...

class Product(webapp.RequestHandler):
    def get(self, productId, version):
        # Get identified product
        # ...
    def delete(self, productId, version):
        # Delete identified product
        # ...
    def put(self, productId, version):
        # Update identified product
        # ...

application = webapp.WSGIApplication(
    [
        ('/API/products', ProductList),
        (r'^/API/products/(\w+)/(\d+)$', Product),
    ],
    debug=True
)

def main():
    # Global initializations
    # ...
    run_wsgi_app(application)

if __name__ == "__main__":
    main()

Code 2: Excerpt of the Python file processing product-related requests

Note that the second code sample shows an ideal situation. In reality, I had to change the verb PUT when updating product definitions because the method self.request.get() cannot extract information from the stream—it does only work for GET and POST verbs. The corresponding client-side code relies on dojo.xhrPost() instead of dojo.xhrPut(). If you know the fix or a work-around, do not hesitate to post a comment ;)

While developing application front-ends, developers should always rely on the MVC pattern to separate the data from user interface, to separate the data flows from the interaction processing. IMHO, organizing the server-side interface as a RESTful API is very clean and efficient. If you use Dojo to build your JavaScript application, you can even rely on their implementation of various RESTful data sources [12] to simplify your work.

Credits: SitePen

Pushing the MVC pattern browser-side has nasty side-effects when too much information are managed by the Model proxy:

• Large data sets consume a lot of memory.
• HTTP connections being a rare resource sometimes unreliable, rescheduling requests (important ones first, non important to be replayed later) or replaying requests (because Microsoft Internet Explorer status reports WSAETIMEDOUT for example) complexify the data flows.
• Fine grain API consumes a lot of bandwidth especiallywhen the ratio data vs envelope is negative).
• Applications have often too few entry points, hiding then the benefit of one URI per resource (intrinsic REST value).
• In highly volatile environments, data synchronization become rapidly a bottleneck if there is no push channel.

So the application performance (response time and memory consumption) should be carefully monitored during the development. If applications with the MVC pattern organized browser-side and relying on RESTful APIs cannot do everything, they are definitively worth prototyping before starting the development of any enterprise application or application for high availability environments.

A+, Dom
--
Sources:

1. Motif definition on wikipedia and online book introducing X/Motif programming.
2. History of Borland Object Windows Library (OWL) on wikipedia and its positioning against Microsoft Foundation Class (MFC) library.
3. FastCGI website, and its introduction on wikipedia.
4. Presentation of the JavaServer Pages (JSP) technology on SUN website.
5. History of XMLHttpRequest on wikipedia, and its specification on W3C website
6. Dojo resources: introduction on wikipedia, Dojo Toolkit official website, Dojo Campus for tutorials and live demos, online documentation by Uxebu.
7. Introduction of the Design Patterns on wikipedia and reference to the “Gang of Four” book (Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides). Specific presentation of the Model View Controller (MVC) pattern on wikipedia.
8. Django : default template language available in Google App Engine development environment.
9. Definition of the Representational State Transfer (REST) architecture on wikipedia.
10. Future post will describe the nature of this project ;)
11. HTTP status codes on wikipedia, and section 10 of the RFC2616 for the full status code list. Don't forget to look at the illustrations of the HTTP errors codes by the artist Adam "Ape Lad" Koford (license CC-by).
12. RESTful JSON + Dojo Data by Kris Zyp, with details on the dojox.data.JsonRestStore introduced in Dojo 1.2.