App Engine 1.3.7 SDK Bugfix Release

Last week we announced our 1.3.6 release with lots of new, exciting features to try out. After it was released, developers dug into all the new functional and discovered a pair of issues that required a quick fix. Today we are releasing version 1.3.7 of the SDK in both Java and Python to fix two issues that shipped with last week’s release 1.3.6.

The first issue was with the Python SDK, which was missing the functionality to assign a namespace based on the Google Apps domain. With this release the function google_apps_namespace() should exist and function as documented.

The second issue we fixed was with the Java SDK. The getServingUrl() function for the new dynamic image resizing service was throwing a SecurityException in the SDK. This issue has now been fixed and this function should work as expected.

Thank you to all the people who reported these issues. Please visit the download page to get the new version of the SDK.

Posted by the App Engine Team

Multi-tenancy Support, High Performance Image Serving, Increased Datastore Quotas and More Delivered In New App Engine Release

Today marks the 1.3.6 release of App Engine for Java and Python. In this release we have made available several exciting new features, relaxed quota and datastore limitations, and added various issue fixes.

Multi-tenant Apps Using the Namespaces API

We are pleased to announce support for multi-tenancy for applications via the Namespaces API. With multi-tenancy, multiple client organizations (or “tenants”) can all run the same application, segregating data using a unique namespace for each client. This allows you to easily serve the same app to multiple different customers, with each customer seeing their own unique copy of the app. No changes in your code are necessary to use this API-- just a little extra configuration. Further, the API is also designed to be very customizable, with hooks into your code that you can control, so you can set up multi-tenancy in any way you choose.

Check out our application examples for Java and Python to demonstrate how to use the Namespaces API in your application. The API works will all of the relevant App Engine APIs (Datastore, Memcache, and Task Queues). Check out our docs for Java and Python to learn more.

High-Performance Image Serving

This release also includes a new, high-performance image serving system for your applications, based on the same infrastructure we use to serve images for Picasa. This feature allows you to generate a stable, dedicated URL for serving web-suitable image thumbnails. You simply store a single copy of your original image in Blobstore, and then request a high-performance per-image URL. This special URL can serve that image resized and/or cropped automatically, and serving from this URL does not incur any CPU or dynamic serving load on your application (though bandwidth is still charged as usual). It’s easy to use, just call the Python function get_serving_url, or the Java function getServingUrl and supply a Blob key (with optional serving size and/or crop arguments), and you can now serve dozens or hundreds of thumbnails on a single page with ease. To enable high performance image serving in your deployed application, you'll need to enable billing.

Custom Error Pages

Since launch, many developers have asked to be able to serve custom error pages instead of those automatically served by App Engine. We are happy to announce today we are supporting static HTML error pages that can be served for you automatically for over quota, DoS, timeout and other generic error cases, that you previously could not control. You can configure custom error handlers in your app.yaml or appengine-web.xml file. Check out the Java or Python docs for more information.

Increased Quotas

We have also continued the trend of lifting some system limitations that have been in place since launch. The Datastore no longer enforces a 1000 entity limit on for count and offset. Queries using these will now safely execute until they return or your application reaches the request timeout limit. Also, based on your feedback, we have raised nearly all of the burst quotas for free apps to the same level as the burst quotas for billed apps. Check out the docs for more information on quota limits.

Other features included in 1.3.6:

• Java developers can now use the same app.yaml configuration file that App Engine uses for Python applications instead of appengine-web.xml (if preferred).
• You can now pause task queues via the Admin Console interface.
• Dashboard graphs in the Admin Console will now begin showing up to 30 days worth of data.
• Content-Range headers are now supported with the Blobstore API.

You can read all about these features in the App Engine documentation for Java and Python. The new versions of the SDK can be found on our downloads page.

In addition to all of the new features, we’ve included several bug fixes which you can read all about in the release notes for Java and Python.

-- Posted by the App Engine team

WebFilings streamlines SEC reporting using Google App Engine

Greetings App Engine Developers!

My name is Dan Murray. I'm a Founder and Managing Director at WebFilings http://www.webfilings.com. In March 2010 my company launched a first-of-its-kind SaaS solution that dramatically improves the process of developing and filing financial reports with the Securities Exchange Commission (SEC). Google App Engine provides the foundation for our backend. Since our launch, my company is experiencing what can only be described as hyper-growth. Everyday, as our team manages the rapid growth of our user community, I am thankful we chose App Engine when we started WebFilings.

Our Challenge

Each quarter, thousands of public and private companies in the US report their earnings to the SEC. The SEC reporting process is complex, time-consuming, and expensive. Reporting requirements and accounting codes are in a state of constant flux. The effects of misreported finances can be extremely costly to a company both financially and in reputation.

Prior to starting WebFilings, my partners and I spoke with dozens of companies and listened to them describe their approach to SEC reporting. While each company has its own unique challenges and methods, the general theme was the same: today's word processors and email just don't cut it for financial reporting. SEC reporting requires more sophisticated collaboration and data management capabilities than current tools provide.

We formed WebFilings almost immediately after these meetings and started building our product. We knew our software would manage a large amount of very granular data for which history and revision control are important. Individual data entities would be interrelated and a change to one may require a cascade change to hundreds, thousands, or more entities. Our product needed rich collaboration capabilities to allow multiple authors to work on a single report simultaneously and communicate effectively throughout the reporting process. Finally, because our software would manage sensitive, pre-release financial information, our product had to be state-of-the-art when it came to information security.

Why Google App Engine?

We started working with Google App Engine right after it launched in 2008. Since we started, our development team has grown from a few guys in a basement to an extensive worldwide organization. From the beginning, the choice of using GAE for our backend was obvious. App Engine provides many benefits to a business building a large-scale, data intensive product on a short timeline.

App Engine gives us a flexible, object-oriented datastore that enables us to iterate in a way that is just not possible with traditional database and ORM software. When combined with the ease of deployment to the cloud, we are able to develop at a rapid pace and deliver timely, incremental solutions for our customers' emerging needs.

In addition, App Engine appspot gives us a secure place for our customers' data. It's been clear from the beginning that the App Engine Team works very hard to plug each and every possible security hole. Large organizations all over the world are switching to Google Apps. Real peace-of-mind comes with Google's reputation for data security. I can think of no other company in the world that takes security more seriously and actually walks-the-walk.

Finally, our costs with Google App Engine have been unbelievably low. In past ventures, I got used to huge startup costs and large expenditures each month for hardware infrastructure and support staff. That just isn't the case with App Engine. We don't need a large IT staff to support our product because App Engine scales automatically. Our costs have been and continue to be orders of magnitude lower than anything I've ever experienced.

How we built our implementation

I had always heard that building a word processor is one of the hardest things you can ever do as a software engineer. Well now I know for a fact that it is. In addition to building a word processor, we set out to create a spreadsheet editor as well. Both applications run in a web browser and are interoperable in a way that enables you to "link" very granular data between them. Add in the ability to share documents and track revisions among a large team of collaborating editors, and building WebFilings software has easily been one of the most challenging endeavors during my 20+ years in software.

Using WebFilings rich editor to file financial reports

Google App Engine has been absolutely key to our success in ramping up development quickly and building our product in a very short timeframe. We built our product using App Engine's python runtime environment. The combination of python and App Engine's datastore enabled us to take an iterative and incremental approach to building our backend.

WebFilings software is based upon a multi-tier, service-oriented cloud mashup as shown below.

WebFilings' architecture

Our presentation tier consists of three end-user applications. Our editor and reader applications are implemented using the Adobe Flex SDK and delivered to the users desktop using Adobe Flash Player 10. Our administration application is AJAX-based and implemented with HTML, CSS, Javascript, and JQuery.

Our editor and reader applications combine a traditional word processor with a spreadsheet editor. Each contains logic that controls the content, layout, and display of formatted text. The foundation of these applications is Adobe’s Text Layout Framework, an extensible, open source, ActionScript library that leverages the text engine in Adobe Flash Player 10.

Our administrative application contains logic that controls CRUD operations for accounts and users. Through our administration application one administers usernames, passwords and password policy, user account access, subscription levels, and various account and document permissions.

Our business tier is developed using Google App Engine. Web services are implemented using python and the django web application framework. Our Flex/Flash-based editor and reader communicate with the business tier by exchanging serialized binary representations of WebFilings business objects through a Django-based web services API. The AJAX-based administration application communicates with the business tier using JQuery. Service requests and responses are sent via HTTPS.

The business tier implements logic in two sub-tiers. The first of these sub-tiers is responsible for basic business logic that provides methods for foundation and often repeated operations such as creation of a new document. The second of these sub-tiers is responsible for business processes that chain together basic business services to accomplish larger, more complex operations such as sharing a users’ changes to a document.

Business service and process methods access data through the App Engine Data Services API. This API provides a python-based abstraction layer consisting of convenience methods that unify the data tier API with security sandbox methods designed to prevent unauthorized data access across Google appspot instances.

WebFilings end-user application interoperates with read-only search services and transient translation services hosted using Amazon EC2. These web services include an XBRL taxonomy search service, and XBRL validation service, an XBRL instance document viewer, and an EDGAR HTML translation service.

The XBRL taxonomy search service, and XBRL validation service, and EDGAR HTML translation service use Apache Tomcat and the Java Servlet API. In addition, the XBRL taxonomy search service uses MySQL and the Hibernate ORM SDK for storage, access, and indexing of the SEC’s public taxonomy data.

The XBRL instance document viewer is a Microsoft .NET Framework-hosted implementation of the U.S. Security Exchange Commission (SEC) XBRL reference viewer.

We've been fortunate that the App Engine Team continues to churn out innovations at a rapid pace. AppStats has helped us identify problem areas and fine tune performance. Task Queues have given us a good solution for processing large amounts of data and helped us reduce request latency. Memcache has provided us up to 10x performance increase for critical areas of our application. We've been able to absorb these innovations quickly without having to re-architect.

Future Apps on App Engine As WebFilings grows, we will continue to expand our product and the web services that support it. When we find opportunities to build new applications there is no doubt that Google App Engine will be at the core.

Please visit http://www.webfilings.com to learn more about WebFilings software. Our team is growing very fast and we're always looking for top software talent. Check out our Careers page If you want a new and exciting challenge with one of the best software teams around.

Posted by Dan Murray, Co-Founder and Managing Director, WebFilings

Practical Report Generation on App Engine

We recently announced the Mapper API, a first step in a broader effort to provide full MapReduce capabilities on App Engine. While we still have some work ahead of us, there’s already a lot that can be done with today’s Mapper API. One area of particular interest is report generation.

Most applications create and maintain large numbers of detail records: entities in data models, transaction history or event logs. In order to glean useful information out of these vast data sets, your application has to iterate over your entities, summarize and breakdown the results. That’s where the Mapper API comes in.

The Mapper API uses the Task Queue to enable your application to rapidly iterate over its data sets, whether small or very, very large. The API takes care of the tedious bookkeeping involved in efficiently scheduling and keeping track of all those tasks. The Task Queue, in turn, automatically ‘pushes’ the work to your app.

Christopher O’Donnell (@markitecht), a technology product designer and developer from Cambridge, Massachusetts, was kind enough to share with us some of the motivations and implementation details behind his own project. In a new guest article, Modern Funnel Analytics Using Mapper, Christopher makes extensive use of the Mapper API to illustrate the process of generating rollup reports. With his approach, he's able to provide both summarized results and drill down capabilities.

If you like Christopher’s article, you should also checkout out the many other interesting articles on the Google App Engine home page. And, if you have an interesting App Engine article or story you'd like to share, let us know. We'd love to hear about it.

Posted by Fred Sauer, App Engine Team

Rapid cloud development using App Engine for the Cycle Hire Widget Android application

Update Oct 2010: It's been several months since we introduced you to Little Fluffy Toys and their exciting Google App Engine story. Since then, their app has continued getting rave reviews. We're happy to let you know that there is a great follow-up to this story from the Android and UI/UE/UX perspective, and you can find it here: http://blog.radioactiveyak.com/2010/10/android-app-surgery-cycle-hire-widget.html

---

This post is another entry in our ongoing series of guest posts contributed by App Engine developers. Today we partner with Little Fluffy Toys Ltd to tell the story of how they were able to learn App Engine (plus Python) and launched their service paired with their Android application in less than a week!

Introduction

Last week, Little Fluffy Toys Ltd (LFT) launched an Android app to help its users find bicycles and rental locations in London. While this story doesn't sound particularly phenomenal, how they accomplished this using Google App Engine (and Android) makes their application and its launch one of the most exciting success stories so far in 2010.

The development team at LFT were able to quickly come up-to-speed on learning a new programming language and development environment in order to build and launch the App Engine backend service for their Android mobile app to the world in less than one week. The executive summary:

• Attended 1-hour Thursday night presentation on Google App Engine (Jul 22)
• Started to learn Python and App Engine on Saturday afternoon
• Launched live service Wednesday, announcing their Android app with an App Engine backend (Jul 28)

Before we get to the good stuff, a brief backgrounder on the project which spawned the application: metropolitan bicycle-sharing systems, specifically London's. Based on the success and popularity of the Paris Vélib' system, the Barclays Cycle Hire scheme originated mid-November back in 2008 from its mayor, a strong cycling proponent.

The system launched on July 30, 2010; however a month before the project reached its completion, a call for apps was given by the mayor seeking independent developers so that there would be a variety of mobile and web apps available by show time. Enter Little Fluffy Toys Ltd which ended up creating the Cycle Hire Widget app for Android. They needed a backend system to manage the data, found App Engine, and the rest was history. Shortly after going live with their app and the launch of London's bicycle rental system, I had a chance to discuss how their project came together with the help of App Engine.

NOTE: Cycle Hire Widget is only available in the Android Market if you are located within the UK. If you wish to view the application from outside the UK, please download and install it from within your Android browser via this link — bear in mind that the distances to your nearest rental/hire location will be ridiculous!

App Engine and LFT

As we mentioned above, the sole purpose of their App Engine app is to receive data from and provide data to the Android app running on users' mobile phones. The App Engine stored data is "global" for all mobile clients out there, and this includes names, locations, and dynamic specifics related to each bike station such as the data found in the app's screenshot below. Take note all the valuable data that is provided in real-time by App Engine:

If you're a bit familiar with App Engine, you would no doubt have heard about it as a platform for web applications, but this is a use case highlighting App Engine for a non web-based server-side application where no part of the app is user-facing save for what gets sent back to the mobile app. While this type of app doesn’t get much press, it’s more common than people think.

I met Kenton Price, the director and chief architect of Little Fluffy Toys Ltd, at a developer event recently, and he seemed to think that App Engine would be the right tool for the Cycle Hire Widget. It turned out to be true(!), and when I asked about LFT's needs and how they were met by App Engine after their successful product launch, here is what he had to say:

"As you know, we were massively against the clock with the launch of the cycle hire scheme, and we needed something we could get going with fast that would effortlessly scale to perhaps tens of thousands of mobile users. App Engine seemed the perfect choice from what we had read of it before the meeting, and after your presentation it was obviously the way to go. Your recommendation to use Python was scary given neither of us knew a thing about it, but then again we only knew Java from Android not from web development so we didn't have the domain knowledge of building Java web services. So we went with Python, and it worked out really well. I'm astounded how we actually delivered this product in a very short space of time when we both have full schedules working on projects for our clients and other demanding outside interests. Particularly satisfying was having a solution that was agile and flexible enough to enable us to display live cycle availability data within hours of it becoming unexpectedly available at the launch, so we were live in the field with real-time data that very same launch morning, a feature our competitors are still struggling to replicate."

Development experience

Reuben Harris, LFT's chief technical officer, is the lead App Engine developer for the Cycle Hire Widget. He had a great experience even though he was new to Python as well as App Engine. What excited him the most, and what was his development experience like? He tells his story here:

"The single coolest thing about this project is that it was possible to go from a state of knowing nothing whatsoever about App Engine or Python (other than the mile-high view) to having a working and useful application inside of eight hours. We're long-time geeks but we're not geniuses. For us to pick up a new language, a new SDK, a new environment, a new way of doing things, and produce anything of value at all in such a short time speaks volumes about the value, potential, and quality of App Engine and Python.

After installing the App Engine SDK, yes, the very first thing I did was your online tutorial. I did "Hello World" to find my feet then continued into webapp, since a clean URL handler with easy ways to get at HTTP variables seemed essential. Then I immediately jumped into learning about data storage. And wow, what an enlightenment that turned out to be! Goodbye SQL, don't think I'm going to miss ya.... :-)

Since the app's purpose is to manage just ~400 simple objects representing Cycle Hire Stations, each of which contains only Plain Old Data types — no object references or anything possibly messy — I felt I knew enough to implement it now, and so I dived in. And it was so easy! I started with a handler to rebuild the datastore from scratch. Then I wrote a "get" type of handler to retrieve information about groups of hire stations (returning the data in JSON). Finally I wrote an "update" handler so that updated information about cycle hire stations could be posted, and that was it. Job done.

One thing that initially confounded me was an HTTP 500 error caused by our "reset" handler exceeding the 30-second request limit. For a while I was ready to despair; HTTP 500s to anyone with much ASP experience usually means a hideous low-level bug somewhere! However, once we discovered the problem, this was easy to fix by splitting the work into multiple requests (/reset1, /reset2, etc.) It's an admin function that only we'd ever be using, so no harm done and no need to work out anything more clever.

I know we've barely scratched the surface of what can be done with App Engine. We've yet to use Memcache, background tasks, batched updates, or anything beyond simple cloud-based data storage. But that simple thing alone seemed then, and still seems, not far short of miraculous. To not have to worry about databases, servers, uptime, upgrades and above all scaling... to not have to think about any of that at all is such an immense freedom. I'm completely hooked on it and am unlikely to go back to my traditional server tools of MySQL and PHP.

To see Reuben's work in action, check out this video demonstrating how to use the Cycle Hire Widget app while roaming the streets of London seeking a bike to rent/hire or park:

Conclusion

Since the launch, the Cycle Hire Widget has gotten rave reviews from CNET, The Guardian, and The Londonist. It has even been featured by the Press Association of the UK and Ireland! One user commented on Android Market: "Can't really think of a way to make it better," a sentiment reflected in its very high feedback rating. It certainly does sound like quite a success. What does the future look like? I asked Kenton about how LFT came about as well as how they're looking to improve their succeeding offerings, and here's what he had to say:

"We formed Little Fluffy Toys Ltd as a vehicle for Android development where we do consultancy work as well as our own stuff like the Cycle Hire Widget and Social Wallpaper. Whilst all custom development enquiries are very welcome, we're also interested in hearing from people or organisations that would like us to customise Cycle Hire Widget for their particular domain, whether it's cycles with availability in another city, coffee shops with opening hours in a geographic region, or dieting group meetings at pertinent times nearby. You name it, there are a gazillion applications for it!"

Well here on the App Engine team, we're happy for Kenton and his team on being able to implement the server-side solution they needed in such a short period of time on App Engine, and better yet, to help out a worthy cause. Google itself is a socially responsible company that applauds efforts like Barclays Cycle Hire, so we're proud that technologies we provide such as Android and App Engine can be used to help make London and the Earth more sustainable!

Posted by Kenton Price & Reuben Harris, Little Fluffy Toys Ltd, and Wesley Chun, App Engine team