Wednesday, July 28, 2010

App Engine and IPv6, Round 2

Back in March, we announced that all App Engine apps hosted on appspot.com would be reachable over IPv6, from ISPs enrolled in the Google over IPv6 program.

Today, we're adding the same service to all domains hosted by ghs.google.com. If your App Engine app runs on your own domain name using Google Apps, then be sure to review our earlier blog post, which discusses the caveats of parsing client IP addresses, and how to use the SixXS Gateway to test your site over IPv6.

Currently, the Google over IPv6 program presents AAAA records only to ISPs within our whitelist. This helps ensure that our services remain reliable, by reducing exposure to clients with broken IPv6 connectivity. As the maturity of IPv6 deployment improves, we expect this whitelist to grow, eventually disappearing as it encompasses the entire Internet.

That said, we know that some of you are fans of IPv6 who would prefer to skip the whitelist, and offer dual-stack service to everyone immediately. To make your domain bypass the whitelist, just change your CNAME pointer from ghs.google.com to ghs46.google.com. Please note that, while ghs46 will cause your App Engine site to receive a larger fraction of its traffic over IPv6, our studies show that it may make your site slow or unreachable for up to 0.1% of clients across the Internet, so use this alternate domain with care.

As before, if you experience any problems with IPv6 serving and App Engine, please report it in the App Engine issue tracker or post about it in the App Engine discussion group.

Tuesday, July 27, 2010

Update on Datastore Performance Status

If you have been running an App Engine application over the last few months, you will have noticed that performance of the Datastore has drastically improved. In fact, it quickly improved after the blog post we wrote in May acknowledging the performance issues. We chose to wait until this point to ensure that we could maintain the performance and quality of service. Thanks to a huge effort from the Datastore engineering team and our Production team, the metrics over the past month have made us confident that we have returned to a level of performance that we are proud to offer. With that in mind, we are now going to re-enable Datastore CPU usage billing.

On Tuesday, August 3rd, we will re-enable charging for Datastore CPU usage.

This does not mean that we are stopping work on Datastore performance and reliability-- quite the opposite, in fact. The issue has forced the team to step back and prioritize our long term roadmap for the Datastore. While we’re happy with our performance now, there are some very large changes to the Datastore in the pipeline that will continue to improve the App Engine platform. These include features such as a more highly replicated Datastore (now part of the roadmap) and improvements to the index building pipeline.

Thank you all for your patience and continued use of App Engine during this time. We’re excited to continue to serve our apps’ continued growth.

Friday, July 23, 2010

Introducing the Mapper API

At Google I/O we announced the Mapper API. Built completely on top of public App Engine APIs today, this API is only the first component of App Engine’s MapReduce toolkit, but can be extremely useful on its own.

The Mapper API can already be of use to many developers who would otherwise need to build their own tool for doing large scale data manipulation. In addition to taking care of the distribution of these jobs over task queues, it provides the ability to store state, batch datastore writes via mutation pools, and ships with an easy to use administrative interface for job management, all optimized for the constraints of App Engine’s dynamic serving environment. Some examples of the types of operations that work with minimal configuration with this tool:

  • Report Generation
  • Large scale migration of entity properties
  • Datastore cleanup
  • Computing statistics and metrics
For an introduction to the Mapper API, watch Mike Aizatsky’s video from Google I/O, where he demonstrates building a source code indexer. The slides can be downloaded here, and the video is below:

The App Engine team has also written a few great articles on how to use the Mapper API.
  • For Python developers, take a look at the Python Mapper API post on Nick Johnson’s blog.
  • For Java developers, Ikai Lan has written a great post about the Java Mapper API, which takes some design cues from Hadoop’s API and includes several examples of common operations such as large scale modification of properties or batch delete.

When you’re ready to jump in and start using the tool, head over to the project homepage on Google Code. You’ll want to check out the “Getting Started” page for the language you’re using:

Happy Mapping!

- Fred, Mike, Ikai, Nick + the App Engine team

Thursday, July 8, 2010

Upcoming App Engine conference appearances

Fresh from our exciting news from Google I/O last quarter (launch of Google App Engine for Business and new product developments), we now look ahead to the mid-summer months. Since I/O, we've already appeared at a number of events including the Google DevFest Australia and Philippines, PyCon AU, and the Malaysia Open Source Conference, with many more coming this month. There are several well-known large conferences (and many smaller ones too) that our team members will be attending and/or speaking at, so hopefully we'll get to meet you at one of them!

In addition to the ones listed above, Google will be hosting more DevFests in Asia and South America later this year. Keep an eye out for specific dates and locations on the Google DevFest home page. There are also 5 larger global Google Developer Day events in the fall. We would love to meet with App Engine developers at a DevFest, Developer Day, or any of the other conferences above!

See you around the world!

Wednesday, July 7, 2010

Upcoming Datastore Downtime

We have two scheduled upcoming maintenance periods for the Datastore:

  • July 7th, 2010 - 5pm-6pm Pacific Time (July 8th, 0:00 GMT) - Later today
  • July 14th, 2010 - 5pm-6pm Pacific Time (July 15th, 0:00 GMT)

As part of our on going Datastore reliability work, we'd like to inform developers that during the period between the two maintenance events listed above, we are expecting that Datastore performance will be impacted and applications will see higher read/write latencies. This maintenance and the higher latency in between are an unfortunate, but necessary step in our long-term plan towards completing the work to address the datastore latency issues (as described in our recent blog post). We recognize the inconvenience to applications and their users caused by this. For this reason, we continue to offer free datastore CPU usage until we are confident we have cleared these issues.

You can read more about the currently scheduled maintenance periods on the notification to the Downtime Notify list. As a reminder, be sure to sign up with the Downtime Notify list for alerts and updates on scheduled (or unplanned) maintenance, and follow @app_engine on Twitter for ongoing announcements.

Wednesday, June 30, 2010

App Engine SDK 1.3.5 Released With New Task Queue, Python Precompilation, and Blob Features

Today we are happy to announce the 1.3.5 release of the App Engine SDK for both Python and Java developers.

Due to popular demand, we have increased the throughput of the Task Queue API, from 50 reqs/sec per app to 50 reqs/sec per queue. You can also now specify the amount of storage available to the taskqueue in your app, for those with very large queues with many millions of tasks. Stay tuned for even more Task Queue scalability improvements in the future.

Additionally, in this release we’ve also added support for precompilation of Python source files to match the same feature we launched for Java last year. For Python, you can now use precompilation to speed up application loading time and to reduce CPU usage for new app instances. You can enable precompilation by including the following lines in your app.yaml file:

derived_file_type:
- python_precompiled

This will start offline precompilation of Python modules used by your app when you deploy your application. Currently precompliation is off by default for Python applications, but it will be enabled by default in some future release. (Java precompilation has been enabled by default since the release of 1.3.1.)

To give you a taste of what this feature is like, we tested this on a modified version of Rietveld (which included a copy of Django 1.0.4 in the app directory, and which did not use the datastore in its base url). The latency and CPU usage results for the initial load of the application, after uploading a new version of the app and requesting the homepage, were:

Before precompilation enabled:
Test 1: 1450ms 1757cpu_ms
Test 2: 1298ms 1523cpu_ms
Test 3: 1539ms 1841cpu_ms
After precompilation enabled:
Test 1: 805ms 669cpu_ms
Test 2: 861ms 702cpu_ms
Test 3: 921ms 803cpu_ms

Of course, any individual app’s performance will vary, so we recommend that you experiment with the setting for your application. Please submit your feedback and results to the support group!

In addition to Task Queues and Python precompilation, we have made a few changes to the Blobstore in 1.3.5 We have added file-like interfaces for reading Blobs. In Python, this is supported through the BlobReader class. In Java, we have implemented the BlobstoreInputStream class, which gives an InputStream view of the blobs stored in Blobstore.

More information on this release can be found by reading the release notes for Python and Java.

PayPal introduces PayPal X Platform Toolkit for Google App Engine


Hello App Engine Developers!
My name is Praveen Alavilli (@ppalavilli) and I work as a developer evangelist for PayPal's X Platform at PayPal.com. I want to take the opportunity to introduce you to our new open source toolkit for Google App Engine that provides an easy way to integrate your Java apps running on App Engine with the new PayPal's Adaptive Payments API. Currently this is a Java toolkit explicitly for App Engine Java, but a Python version will be coming out soon.

Background
The Adaptive Payments API provides a set of core services offered by the PayPal X Open Global Payments Platform to enable developers to embed payments into their applications, services, and platforms. The Adaptive Payments APIs offer several new payments functionality like Split Payments and Preapprovals, that enable developers to implement a variety of monetization models - freemium, subscriptions, pay-per-use, value-added-services, micro-transactions, e-commerce, etc. in their applications built and running on the App Engine. Whether you are building an application for Businesses to process back-end disbursements or payouts to affiliates, or building a social or gaming app for Facebook / Twitter / Open Social, or building a Desktop gadget for premium content, or building a Geolocation app that only helps users find places and people around where they are but even help in transacting them, or building a Web2.0 AJAX app that mashes up content and services, or several more use cases enabled by App Engine, now you can use the PayPal X toolkit to enable payments in them as it fits the needs.

Using the toolkit
Getting started with App Engine toolkit is easy. You can either checkout the source code from svn and import it into your Eclipse project or simply download the prebuilt jar file and include it in your application's class path (/WEB-INF/lib). Similar to other APIs that you might have used, you would need PayPal API Credentials to authenticate your API requests. With the toolkit you simply create an "APICredential" object from one of your application initialization methods (in most cases from your Servlet init() method) and load the API Credentials that you have obtained from the PayPal Sandbox. (Please refer to PayPal's Sandbox guide for more detailed information on how to obtain them).
// Obtain the credentials from your configs 
credentialObj = new APICredential();
credentialObj.setAPIUsername(getServletConfig()
    .getInitParameter("PPAPIUsername"));
credentialObj.setAPIPassword(getServletConfig()
    .getInitParameter("PPAPIPassword"));
credentialObj.setSignature(getServletConfig()
    .getInitParameter("PPAPISignature"));

// setup your AppID from X.com
credentialObj.setAppId(getServletConfig()
    .getInitParameter("PPAppID"));

// setup your Test Business account email 
// in most cases this would be associated with API Credentials
credentialObj.setAccountEmail(getServletConfig()
    .getInitParameter("PPAccountEmail"));

// add required error condition checks
//....

Once the APICredentialObj is initialized successfully, save it in the application's local context so you do not need to reinitialize it on every request.
At its core, the Adaptive Payments API provides 5 generic API methods: Pay, Pay Details, Preapproval, Preapproval Details, Cancel Preapproval, Refunds, and Convert Currency. The toolkit provides the base API Request classes required to make those API calls. To make it even simpler, the toolkit also provides a few functional API wrapper classes that not only abstracts the APIs in terms of the functionality exposed (SimplePay, ChainedPay, ParallelPay, CreateSimplePreapproval, CreatePreapprovalForPeriodicPayments, PreapprovedChainedPay, etc.) but also provides a few exceptions that helps in handling errors more easily than the generic API Responses.

A Simple example
To give you an example, let's say you are building a SaaS model application on App Engine that you charge your customers based on their usage. While on-boarding customers to use your app, you can use the Preapproval API to obtain authorization from your customers to charge them for the app/service based on their usage in the future. In this case you can simply use the 'CreateSimplePreapproval' to create and send a request as below:
try {

  // CreateSimplePreapproval request to setup a simple 
  // preapproval with no Payment Period Set
  CreateSimplePreapproval simplePreapproval = 
      new CreateSimplePreapproval();

  // set the API Credentials object (as given in the code above)
  simplePreapproval.setCredentialObj(credentialObj);

  // starting date in yyyy-MM-dd format
  simplePreapproval.setStartingDate("2010-06-25");

  // ending date in yyyy-MM-dd format 
  // in this case it's for an year from the starting date
  simplePreapproval.setEndingDate("2011-06-25");

  // set max total amount of all Payments
  simplePreapproval.setMaxTotalAmountOfAllPayments(52.00);

  // set max amount for each payment - let's say $1
  simplePreapproval.setMaxAmountPerPayment(1.00);

  // set max number of payments allowed - (52 weeks)
  simplePreapproval.setMaxNumberOfPayments(52);

  // set where to send the user in case of a cancellation
  simplePreapproval.setCancelUrl(req.getRequestURL() +
      "?action=preapproval&cancel=1");

  // set where to return the user after successful approval
  simplePreapproval.setReturnUrl(req.getRequestURL() + "?
    return=1&action=preapproval&preapprovalKey=${preapprovalKey}");

  /* Set other required fields */
  // ... //

  // set memo for user's transaction history
  simplePreapproval.setMemo("Preapproval for GAE Sample");

  // send the request
  PreapprovalResponse preapprovalResponse = 
      simplePreapproval.makeRequest();

  } catch (//...exceptions go here...//) {
  // handle exceptions as necessary

}

In this case, when the 'AuthorizationRequiredException' is thrown, your application would need to redirect the user to PayPal for authorizing the preapproval. The 'getAuthorizationUrl' method takes care of building the PayPal authorization url along with the Preapproval Key returned by the Preapproval API. Once the user authenticates and authorizes the preapproval request on PayPal.com, the user will be redirected back to your 'returnUrl' along with the preapprovalKey, which your application can verify by using the 'PreapprovalDetailsRequest' and store it in it's own app engine data store securely. From that point onwards, whenever the user needs to be charged for their usage of the application, the application can use one of the 'PreapprovedSimplePay' or 'PreapprovedParallelPay' or 'PreapprovedChainedPay' classes to make a payment on behalf of the user. Please refer to the sample apps to understand how to use the other classes provided by the toolkit to make Parallel,Chained or Simple Payments. For eg. the PicMartServlet.java in the PicMart sample app shows how you can make a parallel payment to two receivers at the same time.

Further examples
As mentioned earlier, if you are building a social game where users can buy digital goods (eg. micro-payments) and virtual currencies while playing a game, you can use the same Preapproval API to obtain authorization from them so you could charge their PayPal accounts as and when needed, without requiring them to re-enter their payment information or redirect to PayPal for authorizing the payments.
Other examples of using the toolkit can include:
  • An application that lets merchants or enterprises pay their suppliers, or manage affiliate networks
  • Applications to enable property owners to collect rental payments from tenants
  • With a mult-merchant marketplace, a simple payroll app enables employee salary payments in multiple countries
  • ...

As you can see, the toolkit's support of the Adaptive Payments APIs can easily enable all of these usecases.

Where to find more info
Please look at the sample code provided in the samples directory. The AdaptiveRequests.java class in AdaptiveSample and AdaptiveSampleFnAPI shows how to use the helper classes to send and receive requests. The PicMart sample app provides a simple example of how the Parallel Payment can be used in a Photo Printing app that let's users to buy prints of the pictures from Picasa Album of the photographer. This sample app uses the Picasa APIs to fetch the album and picture information from Picasa.

To learn more about the PayPal X Toolkit for App Engine, please visit: http://code.google.com/p/paypalx-gae-toolkit/ and to learn more about Adaptive Payments, please visit: https://www.x.com/community/ppx/adaptive_payments. You can find a lot of resources for developers like documentation, technical spec, sample apps, code, sdks, technical forums, technical support, etc.. on PayPal X Developer Network web site (https://www.x.com).
Posted by Praveen Alavilli (@ppalavilli), PayPal X Developer Network.