Full-text search on App Engine with Whoosh

Question

I need to do full text searching with Google App Engine. I found the project Whoosh and it works really well, as long as I use the App Engine Development Environement... When I upload my application to App Engine, I am getting the following TraceBack. For my tests, I am using the example application provided in this project. Any idea of what I am doing wrong?

<type 'exceptions.ImportError'>: cannot import name loads
Traceback (most recent call last):
  File "/base/data/home/apps/myapp/1.334374478538362709/hello.py", line 6, in <module>
    from whoosh import store
  File "/base/data/home/apps/myapp/1.334374478538362709/whoosh/__init__.py", line 17, in <module>
    from whoosh.index import open_dir, create_in
  File "/base/data/home/apps/myapp/1.334374478538362709/whoosh/index.py", line 31, in <module>
    from whoosh import fields, store
  File "/base/data/home/apps/myapp/1.334374478538362709/whoosh/store.py", line 27, in <module>
    from whoosh import tables
  File "/base/data/home/apps/myapp/1.334374478538362709/whoosh/tables.py", line 43, in <module>
    from marshal import loads

Here is the import I have in my Python file.

# Whoosh ----------------------------------------------------------------------
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'utils')))
from whoosh.fields import Schema, STORED, ID, KEYWORD, TEXT
from whoosh.index import getdatastoreindex
from whoosh.qparser import QueryParser, MultifieldParser

Thank you in advance for your help!

Answer 1

The marshal module is not supported on app engine. It is there, but it is empty. That marshal is working as normal in the dev. environment has been registered as an issue.

See the documentation.

You could try the following to monkeypatch the marshal module. Put the following code before you do any other imports:

import pickle
import marshal
marshal.loads = pickle.loads
marshal.dumps = pickle.dumps # I assume it needs dumps also

I have not tried this, so I have absolutely no idea if it will work! Also be aware that pickle loads/dumps is slower than marshal loads/dumps.

Answer 2

You could probably solve your problems by downloading and using Whoosh-Appengine, the Whoosh version that's specifically targeted to working with Google App Engine.

Answer 3

Another idea.. skip the problem entirely. I made a mashup of Google Base using the GData APIs. Works like a charm, and helps to conserve your CPU overhead.

There's a couple links in that blog post (Don't judge me, judger) of simple sites I made using this approach.

Answer 4

This is an official example about implementing full text search: http://code.google.com/p/guestbook-example-appengine-full-text-search/

I'm currently reading through it as I'm in the need of implementing it, maybe it will help others also.