What's your experience developing on Google App Engine?

Question

Is GQL easy to learn for someone who knows SQL? How is Django/Python? Does App Engine really make scaling easy? Is there any built-in protection against "GQL Injections"? And so on...

I'd love to hear the not-so-obvious ups and downs of using app engine.

Cheers!

Answer 1

The most glaring and frustrating issue is the datastore api, which looks great and is very well thought out and easy to work with if you are used to SQL, but has a 1000 row limit across all query resultsets, and you can't access counts or offsets beyond that. I've run into weirder issues, with not actually being able to add or access data for a model once it goes beyond 1000 rows.

See the Stack Overflow discussion about the 1000 row limit

Aral Balkan wrote a really good summary of this and other problems

Having said that, app engine is a really great tool to have at ones disposal, and I really enjoy working with it. It's perfect for deploying micro web services (eg: json api's) to use in other apps.

Answer 2

GQL is extremely simple - it's a subset of the SQL 'SELECT' statement, nothing more. It's only a convenience layer over the top of the lower-level APIs, though, and all the parsing is done in Python.

Instead, I recommend using the Query API, which is procedural, requires no run-time parsing, and makes 'GQL injection' vulnerabilities totally impossible (though they are impossible in properly written GQL anyway). The Query API is very simple: Call .all() on a Model class, or call db.Query(modelname). The Query object has .filter(field_and_operator, value), .order(field_and_direction) and .ancestor(entity) methods, in addition to all the facilities GQL objects have (.get(), .fetch(), .count()), etc.) Each of the Query methods returns the Query object itself for convenience, so you can chain them:

results = MyModel.all().filter("foo =", 5).order("-bar").fetch(10)

Is equivalent to:

results = MyModel.gql("WHERE foo = 5 ORDER BY bar DESC LIMIT 10").fetch()

Answer 3

Google App Engine doesn't use an actual database, and apparently uses some sort of distributed hash map. This will lend itself to some different behaviors that people who are accustomed to SQL just aren't going to see at first. So for example getting a COUNT of items in regular SQL is expected to be a fast operation, but with GQL it's just not going to work the same way.

Here are some more issues:

http://blog.burnayev.com/2008/04/gql-limitations.html

In my personal experience, it's an adjustment, but the learning curve is fine.

Answer 4

A major downside when working with AppEngine was the 1k query limit, which has been mentioned in the comments already. What I haven't seen mentioned though is the fact that there is a built-in sortable order, with which you can work around this issue. From the appengine cookbook:

def deepFetch(queryGen,key=None,batchSize = 100):
  """Iterator that yields an entity in batches.

  Args:
    queryGen: should return a Query object
    key: used to .filter() for __key__
    batchSize: how many entities to retrieve in one datastore call

  Retrieved from http://tinyurl.com/d887ll (AppEngine cookbook).
  """

  from google.appengine.ext import db

   # AppEngine will not fetch more than 1000 results
  batchSize = min(batchSize,1000)

  query = None
  done = False
  count = 0

  if key:
    key = db.Key(key)

  while not done:
    print count
    query = queryGen()
    if key:
      query.filter("__key__ > ",key)
    results = query.fetch(batchSize)
    for result in results:
      count += 1
      yield result
    if batchSize > len(results):
      done = True
    else:
      key = results[-1].key()

The above code together with Remote API (see this article) allows you to retrieve as many entities as you need.

You can use the above code like this:

def allMyModel():
  q = MyModel.all()

myModels = deepFetch(allMyModel)

Answer 5

At first I had the same experience as others who transitioned from SQL to GQL -- kind of weird to not be able to do JOINs, count more than 1000 rows, etc. Now that I've worked with it for a few months I absolutely love the app engine. I'm porting all of my old projects onto it.

I use it to host several high-traffic web applications (at peak time one of them gets 50k hits a minute.)

Answer 6

My experience with google app engine has been great, and the 1000 result limit has been removed, here is a link to the release notes:

app-engine release notes

No more 1000 result limit - That's right: with addition of Cursors and the culmination of many smaller Datastore stability and performance improvements over the last few months, we're now confident enough to remove the maximum result limit altogether. Whether you're doing a fetch, iterating, or using a Cursor, there's no limits on the number of results.