problem with running jpype with mod_python

Question

Hi,

As Python's urllib module is too slow, I'm using Java code wrapped with JPype in my web site. When I tested my web site with Django web server, there was no problem. However when I switched the web server to apache2 + mod_python, following error occurs. I googled many times but couldn't find the answer. Is there any solution to the error?

MOD_PYTHON ERROR
ProcessId:      4831
Interpreter:    'localhost'
ServerName:     'localhost'
DocumentRoot:   '/home/www/mysite'
URI:            '/javamodule.py/'
Location:       '/'
Directory:      None
Filename:       '/home/www/mysite/javamodule.py'
PathInfo:       '/'
Phase:          'PythonHandler'
Handler:        'django.core.handlers.modpython'

Traceback (most recent call last):

  File "/usr/lib/python2.6/dist-packages/mod_python/importer.py", line 1537, in HandlerDispatch
    default=default_handler, arg=req, silent=hlist.silent)

  File "/usr/lib/python2.6/dist-packages/mod_python/importer.py", line 1229, in _process_target
    result = _execute_target(config, req, object, arg)

  File "/usr/lib/python2.6/dist-packages/mod_python/importer.py", line 1128, in _execute_target
    result = object(arg)

  File "/usr/lib/pymodules/python2.6/django/core/handlers/modpython.py", line 228, in handler
    return ModPythonHandler()(req)

  File "/usr/lib/pymodules/python2.6/django/core/handlers/modpython.py", line 183, in __call__
    os.environ.update(req.subprocess_env)

  File "/usr/lib/python2.6/os.py", line 486, in update
    self[k] = dict[k]

  File "/usr/lib/python2.6/os.py", line 471, in __setitem__
    putenv(key, item)

Answer 1

Another solution for your original problem: find other ways to get faster url retrieval.

httplib2 might already be a good solution: no problem to get it working as it's just a python library, but support for Keep-Alive connections can speed up things a lot, plus the caching support will help too (but only if you are often requesting the same url of course). And it's very easy to use.

If that doesn't get you far enought: PyCurl, the python binding for libcurl is probably the most obvious choice (there are also wrappers to make PyCurl easier to use, as PyCurl is apparently a bit low-level).

Anyway, just to say there are options that do not require java, that are easier to get working (and that will be probably end up faster as well)

Answer 2

Thanks, Steven. I have also tried httplib2 and pycurl. I even tried libcurl and c socket. However they didn't help much in my case, which requires very frequent short queries. I don't know why but java HttpURLConnection was really fast in my case (even than c).

As I am not a network expert, I don't know why this happens. Does anyone know why java shows better performance than c? And any good ideas for very frequent short queries? Thanks in advance.