Datastore Performance Growing Pains

As many of you know, App Engine’s Datastore performance has been seriously degraded over the last few weeks. In addition to May 25th’s 45 minute Datastore outage, applications have seen an increased latency and thus errors as a result of timeouts. As a rough estimate, we have seen Datastore latency increases of around 2.5x. Comparing April 1st to June 1st on the Status Site shows this trend. These issues in turn have caused problems with other parts of the App Engine service that also rely on the Datastore, such as Task Queues and the Admin Console.

There are a lot of different reasons for the problems over the last few weeks, but at the root of all of them is ultimately growing pains. Our service has grown 25% every two months for the past six months. To accommodate this growth, App Engine has expanded, both increasing our computing footprint and moving between Google datacenters to accommodate our growing size. Although we have moved and expanded, it hasn’t been enough-- and performance has suffered.

We want you to know we are taking the current problems with the Datastore very seriously. We have put other Datastore projects on hold to dedicate more people to accelerating improvements to Datastore performance, and to our datacenter configuration. We expect the Datastore may still have a few issues for the next two weeks, as we respond to the problem. After that point you should begin to see rapid improvements in performance, back to normal levels. Until that point, however, we believe Datastore performance may continue to be at a level that we feel is unacceptable.

So until performance has returned to a state we are proud of, and that you are satisfied with, your applications will not be charged for Datastore CPU costs effective on your May 31st bill. When we are convinced we have returned Datastore performance to a level we consider acceptable, we will give you a 7-day notification on this blog before we begin charging again (Note: Your Datastore CPU budget will still control your app’s CPU usage though you will not be charged for the Datastore CPU. So please don’t turn your budget to $0 during this period).

We hope this gives more clarity into the current issues you’ve seen with your application and we sincerely appreciate your patience as we work through them.

Thank you

The App Engine Team

Update: Clarified which Datastore charges will be disabled