Hidden limitations of Google App Engine?

Question

I've been looking into writing a web app that will run on Google App Engine, but before I commit myself to the platform I'd like to know what, if any, limitations there are. I'm aware of the basic CPU/bandwidth restrictions that Google places on the free service, but I'm wondering more about development restrictions like how BigTable compares to a standard relational database and what Python libraries aren't available on the GAE platform (and what alternatives Google provides).

Basically I'm looking for any hidden roadblocks before I commit to the platform. Thanks for your help!

Answer 1

I explored Google AppEngine for my own amusement a while ago, making a lunch 8ball in the process.

http://blurry-lunch.appspot.com/

The system is very easy to work with, and to hit the ground running.

Limitations I saw in Bigtable mostly revolved around dataset size and access time. Part of the application I was making would randomly choose a location out of a list of locations, to due this I am pulling out the complete list of locations, then selecting a random element in python. As the default index is a non-linear GUID, and I didn't bother to setup a separate attribute in the object, as to find the next available id in the system isn't how DataStore is designed.

The problem is, if you need linear access to a massive amount of records to perform an operation, you could run into a limit where your request takes too long, but what data set would cause such a delay isn't clearly defined, as google's systems are massively clustered.

As far as external Python Libraries, you should be fine, as long as the calls they make are python based. You will have to bundle them yourselves into the directory structure that gets uploaded for deployment.

You should be aware that you will be locking your self to their platform, as there is no production ready system that supports their API. They have a standalone appserver for debug purposes, but it certainly isn't suitable for an actual deployment.

Another thing is that GAE is still in beta, with no commercial support options, and you could not run a wildly successful web application currently without a commercial plan. The limitations are too low to even survive a slashdotting for static content.

EDIT: Of course this is all as of Feb 19th, 2009. This could all change wildly, or GAE could even be turned off.

Answer 2

I think your biggest obstacle (but definitely not a limitation) is overcoming the relational mindset that has become mainstream in the industry. The relational model has its place in this world, but it doesn't solve all problem domains, and you can't depend on it in GAE.

What this means in more practical terms is that you will have to design/architect your applications differently. In several scenarios the typical and norm you have been acusstomed to will not apply. Perhaps one of the most obvious examples is that of shared counters. Anyway, all this is hard to see until you have concrete examples. However for starters, I have compiled a playlist of some of the most helpful App Engine sessions that were presented at Google I/O last year. I would advise you to see them as they do a great job at helping one understand how the platform works.

Update: Just recently on Google App Engine's Blog they describe non-relational databases. I think this will help to give more context.

Answer 3

There are a lot of constraints in the google app engine and it is quite difficult to list them all.

A good start for you might be to check the list of most voted feature requests and figure out if your project can work without those: http://code.google.com/p/googleappengine/issues/list. There are other small issues related to the dashboard and inconsistencies between the SDK and the public application, but most of these can be avoided.

One thing that I have found quite annoying is the fact that there is no 'blueprint' for building apps in GAE. Basically, even for quite simple web apps you'll find out that there isn't a known path to decide what is the best way to structure your data and get the best performance out of it, nor are there simple ways to profile and understand how to circumvent these.

Building an app deployable on GAE is a fundamental shift from the traditional way of writing apps and while it is exciting, I do think that it might require more effort on your side.

./alex

PS: I'm currently working on a GAE based project, that is deployed here: http://the.dailycloud.net

Answer 4

There are many limitations that are not marketed by Google, but you'll hit them sooner or later ;). One of them was just posted at http://stackoverflow.com/questions/572780/cpython-internal-structures.

Many limitations can be overcome by changes in algorithms, that will do good for your application anyway. For instance, they recently raised timeout from 10 sec to 30 sec for total request processing (you can spend that time in many ways that doesn't consume other resources, the simplest is querying external system). Changes you do to your application to fit into 30 seconds will make your app better!

And similar approach applies to many other limitations that are there. Try, push to limits, and see if it fits you. Good luck!

Answer 5

There are a number fo limitations, though it is a promising product.

The most ironic of them is that GAE doesn't support TEXT SEARCH in its database Api.

I thought google was all about search, and mostly text search.

Answer 6

Performance will surprise you. GAE is optimized for many tiny queries and you get warned if a query takes any CPU time at all. You get 6.5 (at last check) free hours per day, but it's a mystical number and you should test.

You'll find that time as you measure it doesn't relate to the CPU or datastore CPU time, because (for example) under the covers there might be multiple machines updating indexes during deletes/updates. Some users have found huge CPU usage when uploading bulk data - many hours of usage for e.g. 20 min of real time.

Your Java instance might need to be powered-up if it hasn't been hit in (I think) 20 minutes. The benefit is that they can pass their smart management on to you as cheaper costs, but it does mean you'll experience a short delay, and see a high CPU warning on the first request in a while.

For many cases, Python datastore access is faster than Java JDO. You'll likely find that using the low-level API for Java faster.

Some developers seem to have experienced more datastore errors thank you would expect (around 0.4-1% maybe?). I haven't yet.

Answer 7

I have identified/researched following limitations so far.

• no HTTPS for custom domains. Only for your-app-id.appspot.com domains
• no streaming and long term connections
• a web request must respond in 30 senconds otherwise GAE throw DeadlineExceededException
• application will not work with your naked custom domain. i.e http://yourdomain.com
• only HTTP and HTTPS. Client can not connect to GAE through FTP

Answer 8

I agree with other users.
Having developed a web application with Gae, i would add :

1. You have a 10 seconds max Deadline on download with UrlFetch
2. 30 Seconds Deadline for each process (tasks or requests..whatever)
3. 1 Megabyte max for single put()
4. API currently offers a fairly limited functionality for text-search
5. Portability
6. Deploy environment does not replicate the same restrictions of the Production (this can be painful for testing. Cfr. "works on my machine")

I've found more pros than cons though.

Answer 9

In addition to the other answers, there is a maximum of 5000 indexed properties per entity (source: http://www.youtube.com/watch?v=AgaL6NGpkB8&feature=player_embedded#!) but it really seems to be a limit of 5000 indexed property values per entity.

I summed up more limits of the datastore here http://code.google.com/p/xydra/wiki/AppEngine