Developer Insights: Building scalable social games on App Engine

Today’s guest blogger is Hernan Liendo, CTO of Zupcat, developer of social games played by millions of people worldwide.  Hernan shares his team’s experience using App Engine to build RaceTown, a multiplayer racing game.  

Choosing a cloud service provider

RaceTown is one of Zupcat’s most popular games; it has almost 900,000 monthly unique users, opens more than 40,000 connections via the Channel API per day, processes more than 15,000 queries per second and delivers terabytes of content everyday.  When deciding our architecture, we took into account several unique requirements of social games:

• High uptime
• Short loading time
• Flexibility to deal with social network API changes
• Ability to manage thousands of players, concurrently, from all over the world
• Adjustment to capabilities and performance issues on different users’ computers
• Ability to measure user actions to constantly improve the user experience
• Hosting and delivering quality, beautiful game art
• Complex game domains and algorithms: such as enemy adaptable performance, path finding, and 2D and 3D rendering

App Engine addresses these complicated issues.  It provides few tracerouting hops from almost anywhere in the world, great uptime, automatic scalability, no need for infrastructure monitoring and a reasonable price for content delivery.

Implementing App Engine

The diagram above shows a simplified view of our game architecture. We’ve discovered that App Engine is good to use not only as a game backend server, but also as a metrics server and content delivery network.  In addition, we periodically synchronize game state and retrieve data to and from the server.  

The App Engine Datastore is great because it has high availability and easily handles hundreds of millions of rows of data, which is important for social games.  For example, we  can easily scan the Datastore to present high score information and gamer stats to the user.  Additionally, because gamers tend to spend lot of time during a game session, we’ve found it’s helpful to cache game data. Using Memcache, we have significantly reduced Datastore API calls and lowered users’ waiting time.

Another tip for App Engine developers - although App Engine API failures are uncommon, you must be sure to write proper retrying code to minimize the possibility of exposing users to an application crash.  RaceTown performs almost a hundred million operations daily, and proper client side retrying algorithms have enabled us to reduce failure rates to very low levels.

Final thoughts

I believe that today there is no technology that matches App Engine.  You can run your code and store your data in the very same servers that Google uses.  Migrating your applications to this technology means you have to start thinking in a cloud-centric way and reinvent your architecture to stop working inside a relational database and classic clustered web server.

If you can achieve this, your products will be delivered using the same infrastructure that Google uses, without a huge corporate budget!

- Contributed by Hernan Liendo, @hernanliendo of Zupat, @zupcat