Monday, April 28, 2008

Tips on writing scalable apps

Google App Engine makes writing scalable apps easier, and a well-designed app should be able to grow automatically from one user to millions. What does it mean for a Google App Engine application to be "well-designed," though? Here are a few simple tips that, if you design with them in mind, should help ensure your App will stand up with grace and aplomb, even under heavy load.
  • Avoid contention on datastore entities. If every request to your app reads or writes a particular entity, latency will increase as your traffic goes up because reads and writes on a given entity are sequential. One example construct you should avoid at all costs is the gobal counter, i.e. an entity that keeps track of a count and is updated or read on every request. There are some interesting ways of simulating this behavior that don't require reads/writes on every request, and we'll talk about a handy way to cache entities for reads below.
  • Avoid large entity groups. Any two entities that share a common ancestor belong to the same entity groups. All writes to an entity group are sequential, so large entity groups can bog down popular apps quickly if there are a lot of writes to that group. Instead, use small, localized groups in your design.
  • Write sparingly. Writes are more expensive than reads; keep this in mind when designing your data model. If you can avoid a write--it's best to do so.
  • Define a main() function for code reuse. Instances of your app are kept running for a certain period after each request, so there's a chance that any request will hit an already-running app. Apps that define main as in the following example:

    def main():
       application = webapp.WSGIApplication(_URLS, debug=True)

    if __name__ == '__main__':
    can reuse global variables, so although there's no guarantee that any given request will hit a running app instance, you can take advantage of this to non-deterministically cache entities and other expensive values. This works simply because App Engine will detect if main() is present in an already-loaded script and call it instead of re-importing.
  • Profile your code. You can use the Python profiler to profile your code's performance.
Do you have tips for scaling web apps on App Engine? Post them in our Google Group!

Wednesday, April 23, 2008

Google App Engine + Google Data APIs: A Match Made on the Web

Don't you think it'd be cool if you could blog about a party, add it to your calendar and invite all your friends in one fell swoop? After the party, how about viewing photos and videos from the event in the same application? The Google Data APIs make all this possible, and your Google App Engine apps can use feeds from Google services to display and update information for a particular user stored by these services.

The Google Data APIs team just released an update to the Google Data Python client library which includes support for Google App Engine. With this library, your app can now use AuthSub to gain access to your users' feeds.

Jeff Scudder, member of the Google Data APIs team and the main contributor to the Google Data Python client library, has written an article that takes you through all of the steps necessary to use Google Data feeds in your Google App Engine application. He's even added his sample application to our Google App Engine sample code project.

Try out it, and let us know what you come up with!

Monday, April 21, 2008

Google App Engine Hack-a-thons!

In May, Google will be holding two free Google App Engine developer events: one in New York and one in San Francisco.

Learn about Google App Engine
The events will include training on the major features of Google App Engine, including data modeling, the App Engine APIs, some aspects of Django, and how to mashup App Engine with other web services. Google Engineers and Product Managers will be on hand to help and answer questions throughout the day.

As well as covering the Google App Engine SDK, we will show you how to deploy and manage your own applications on Google App Engine servers. For people who don't already have a Google App Engine account, we will be inviting people who come to the hack-a-thon to develop on App Engine.

Build With Us, or Build Your Own
We will use the information shared at the event to build up a working Google App Engine application throughout the day, which you can code along with us. On the other hand, if you already have a great idea for what to build, bring that idea along. Even better, bring along anything you can prepare ahead of time (sketches, designs, web page mock ups, etc.) and use the time and information provided to develop your idea into a working application, then share it with the world.

What Do I Need?
We will provide facilities, power, food and refreshments and experts to help you learn to use Google App Engine and write your application. Just bring your laptops, ideas and enthusiasm to complete the mix. We will assume some basic skills and preparation for the event, and these details will be provided soon.

When and Where?
The first event will be in New York, on May 7th from 10am to 6pm. The second will be in San Francisco on May 16th from 10am to 10pm. Space is limited so sign up now!

New York Hack-a-thon
San Francisco Hack-a-thon

Friday, April 18, 2008

Google App Engine Helper for Django

Since the launch of Google App Engine one of the most common requests has been for better integration with the popular Django framework.

We've been working hard and we've made available a helper that eases the process of developing a Django project to run on Google App Engine.

The helper will initially provide you with:
  • The ability to use most commands
  • A BaseModel class that appears the same as the standard Django Model class.
  • The ability to serialize and deserialize model instances to JSON, YAML and XML.
  • Access to Django's test framework with a test datastore and support for fixtures.
We are releasing the helper under the Apache 2.0 license. We look forward to working with the Django community to add further functionality as time goes on.

You can download the helper from Issues and Feature Requests should be added to the tracker within that project.

Monday, April 14, 2008

Getting your data on, and off, of Google App Engine

One upcoming feature of Google App Engine that's worth mentioning is large-scale data import and export. We know that many of you have large apps that already run on other platforms, and getting all of that data into the Datastore API is a challenge. And, conversely, we also know that a lot of you want to make sure it's easy to move off Google App Engine in the future, in case you need to switch to a different platform.

With Google App Engine, you own all the data in your app. As stated in our terms, you always have the right to get your data out of Google App Engine at any point. We wouldn't have it any other way.

To that end, we would like to solicit feedback on what form you would like the data exporter feature to take. We've started a thread in the discussion group about this, and we'd love to hear what you think. XML output? CSV transform? AtomPub? RDF? Let us know!

Friday, April 11, 2008

We're up and running!

Google App Engine has been live for several days now and we're thrilled with the response we've gotten so far! We filled our initial batch of accounts within a few hours after Monday night's Campfire One. And today, we've just invited another 10,000 developers to try it out.

We're excited to see the beginnings of a developer community forming in our Google Groups! We're actively reading your posts and responding with help where we can, and it's great to see some of the more experienced members of the community start helping others out too!

We've also seen a lot of feature and bug reports on our Issue Tracker, including requests for Ruby, Java, Perl, PHP, and Fortran. Here are are some of the general areas we're focusing on right now:

  • Support for more languages. We're obviously huge Python fans, but we recognize that there are other great languages out there that developers use to build web applications.
  • Support for offline processing. Right now Google App Engine is great for web apps that do all of their processing in response to user requests, but what about apps that need to perform scheduled tasks or larger-scale data migration? We'd like to support those apps too.
  • Support for large files. Google App Engine currently imposes a limit of 1MB on all requests, both inbound and outbound. We're interested in providing efficient support for much larger uploads and downloads.
  • Billing for additional quota. During the preview release period, all apps are restricted to a set of free resource quotas. Although Google App Engine will always be free to get started, we plan on allowing developers to purchase additional resources in the future, while paying only for what they use.

Judging by the the feedback we've gotten so far, these are areas you're interested in as well--we'll keep you posted. Keep letting us know what you think about the Google App Engine and where you think it should be headed!

Monday, April 7, 2008

Introducing Google App Engine + our new blog

At tonight's Campfire One we launched a preview release of Google App Engine -- a developer tool that enables you to run your web applications on Google's infrastructure. The goal is to make it easy to get started with a new web app, and then make it easy to scale when that app reaches the point where it's receiving significant traffic and has millions of users.

Google App Engine gives you access to the same building blocks that Google uses for its own applications, making it easier to build an application that runs reliably, even under heavy load and with large amounts of data. The development environment includes the following features:

  • Dynamic webserving, with full support of common web technologies
  • Persistent storage (powered by Bigtable and GFS with queries, sorting, and transactions)
  • Automatic scaling and load balancing
  • Google APIs for authenticating users and sending email
  • Fully featured local development environment

Google App Engine packages these building blocks and takes care of the infrastructure stack, leaving you more time to focus on writing code and improving your application.

Today's launch is a preview release -- we're by no means feature-complete, and we're giving you early access because we really want your feedback. This preview of Google App Engine is available for the first 10,000 developers who sign up, and we plan to increase that number in near future.

During this preview period, applications are limited to 500MB of storage, 200M megacycles of CPU per day, and 10GB bandwidth per day. We expect most applications will be able to serve around 5 million pageviews per month. In the future, these limited quotas will remain free, and developers will be able to purchase additional resources as needed.

If you'd like to try it out, sign up for access to the preview release and then download the SDK to get started. The best place to give us feedback is in the discussion group. Future announcements from us will appear there and/or this blog.