I work for Norex, a web development company in Halifax, Nova Scotia. As part of our sponsorship of the 2009 ICF Canoe Sprint World Championships in Halifax (Canoe '09), we developed an application to deliver real time race results to standard and mobile web browsers. Thanks to the ability to rapidly develop and deploy a scalable application on Google App Engine, and to do so live during the event, what began as a small experiment became a huge success for Norex.
Our goal was to import instant results (directly from the timing system FinishLynx), upcoming race information, athlete bios, teams, and past race results from a system used by the race organizers, and to reformat that data for distribution to web browsers. We chose to develop in Google App Engine (with Django and appengine patch) and iUI (iPhone User Interface) frameworks. In fact, this was our first time deploying an app using any of these technologies! While we had a few weeks to build the prototype, we also had to deal with the added challenges of a varying data format and special cases that could occur during live races. As with many software projects, these specifications were not provided on a regular or consistent basis from the race organizers, so we had to anticipate the unknowns and work to adjust to the conditions at the time.
To give you a taste for just how agile we were able to be on App Engine, here are some events during the race which we were able to cope with:
- After the first day, the closed-circuit TV system which the race organizers planned to use for the event was down. Before it was back up and running, we were able to deploy a new view appropriate for display on the TVs.
- By the end of the first day, the application had handled 840,000 requests at a peak of 60 requests/second. This was already several orders of magnitude higher than we had expected. At the start of the second day, we reached 10% of our CPU quota before 7:45am and were processing 100 request/second. While we had originally cached only the main pages, with a few lines of code, we extended our caching strategy to cache every possible page request, a different version for each web platform, and re-deployed during the day's races.
- The rules of the events changed at 2pm on day 3 of the event to allow crew changes. By 4pm, I had reworked the models and views to support the change.
- At 11am on the final day of races, someone in the control booth manually deleted inaccurate records, but this created inconsistencies in the model causing errors. By 11:05am, we had deployed a patch to stop the errors, and by 1pm had deployed a version which gracefully failed when referenced objects were inadvertently deleted.
- Every night we deployed new views of the data as requested by race organizers, participants, and spectators.
We found that new deployments were possible (and quick) because we could stage versions of the application to the App Engine servers, and test these staged versions on live data. Switching from the live version to the staged version and back again takes seconds, thus we could paddle as confidently and rapidly as the racers.
The International Canoe Federation was so impressed by the dependability and versatility of the solution, that they elected to replace their usual channels with our application to serve up the official results for news organizations and to Sport Federation sites all over the world, and to be used internally for calculating medal counts.
All in all, we saw over 1,000,000 page views from 93 countries around the world, and experienced incredible stability and scalability from Google App Engine even when we were spiking 350 requests per second during the finals. It was a big win for Norex, and App Engine has proven itself to be a serious contender for developing scalable web applications. Thanks so much to the Google App Engine team for providing such an outstanding product!
Link to: ZAP Results Application
No comments:
Post a Comment