How can one perform full text search in Google App Engine?

Question

It's a simple question, but I haven't found the answer anywhere. Thoughts and input appreciated.

I'm using Django, too, for what it's worth. :)

Cheers.

Answer 1

With Java GAE, you could use Compass, but that won't help with Django. For Python, Bill Katz offers one solution -- open source -- and these guys offer a Django-specific approach which, however, is free only for non-commercial applications (i.e. if your app makes money they want you to pay for their full-text search). I have no real-world experience with either of these solutions so I can't really give well-grounded recommendations, but from what one can see with just a little playing around they seem quite useful.

Answer 2

Issue 217 Bill Katz released a package to deal with and http://gae-full-text-search.appspot.com/ is available alternatively, levensthein is a another match measure

Answer 3

For now, the real answer is that there is no real full-text search on Google App Engine. The solutions provided by the other answers here are fine for toy data sets, but do not scale to anything more than O(10000) documents or so. Google will have to provide search as an infrastructural feature of GAE. See the feature request for (mostly superfluous) discussion.

Answer 4

An overview of the Python App Engine searches that I am aware of:

Google did add a cut down search using SearchableModel although that has limitations (5000 indexed word limit, String property only not Text):

• http://groups.google.com/group/google-appengine/browse_thread/thread/f64eacbd31629668/8dac5499bd58a6b7?lnk=gst&q=searchablemodel

Or as another posters have pointed out there are these options:

The Quick and simple text search:

• http://www.billkatz.com/2009/6/Simple-Full-Text-Search-for-App-Engine

This product which has a fairly comprehensive free version and a more extensive commercial version:

• http://gae-full-text-search.appspot.com/customers/download/

---

I've read that Google do have a project to bring full text search to App Engine although this is not scheduled to happen any time soon

---

I'd really like to see a comparison of the various searching frameworks and see how they stack up to each other. Does anyone know of any report like this?

Answer 5

@ The Quick and simple text search: http://www.billkatz.com/2009/6/Simple-Full-Text-Search-for-App-Engine

---

this solution did not work for me - and looking at the limitations below, it is unlikely to be useful for real use cases.

1. It uses StringListProperty to store phrases which has a limitation of 500 characters.
2. It does not work with the standard query filters.

Answer 6

You should be able to adapt Whoosh! to write in the datastore instead of on disk. It's a pure python full-text search engine. It's not as fast or full-featured as Lucene, but it should run on GAE without too many modifications.