python imaplib ssl error using celeryd queue

Question

I'm having a problem using imaplib on python 2.6 with the latest django svn. I want to download imap emails in a queue (using celeryd). I'm able to connect/download emails from the command line, but when i offload the task through django to celeryd i get this error: "SSLError: [Errno 1] _ssl.c:1325: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number".

Imaplib docs don't mention how to specify a version of SSL. I'm trying to pull emails from gmail. I don't understand why offloading the task to a queue using celeryd would cause the task to fail. Any help would be much appreciated.

Edit: here is a stack trace:

File "/usr/lib/python2.6/imaplib.py", line 643, in select typ, dat = self._simple_command(name, mailbox)

File "/usr/lib/python2.6/imaplib.py", line 1059, in _simple_command return self._command_complete(name, self._command(name, *args))

File "/usr/lib/python2.6/imaplib.py", line 889, in _command_complete typ, data = self._get_tagged_response(tag)

File "/usr/lib/python2.6/imaplib.py", line 990, in _get_tagged_response self._get_response()

File "/usr/lib/python2.6/imaplib.py", line 907, in _get_response resp = self._get_line()

File "/usr/lib/python2.6/imaplib.py", line 1000, in _get_line line = self.readline()

File "/usr/lib/python2.6/imaplib.py", line 1170, in readline char = self.sslobj.read(1)

File "/usr/lib/python2.6/ssl.py", line 136, in read return self._sslobj.read(len)

SSLError: [Errno 1] _ssl.c:1325: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Edit: Here is the task i'm trying to run, where imap_parser is a module that wraps imaplib and loads emails into my db.

class DumpIMAPData(Task):
    def run(self, user, username, password, imap_address, **kwargs):
        logger = self.get_logger(**kwargs)
        celery.log.redirect_stdouts_to_logger(logger, loglevel=None)
        #imap_address is e.g. 'imap.gmail.com'                                  
        parser = imap_parser.IMAPFetcher(imap_address, username, password, user\
)
        parser.load_all_emails()
    return True

I have noticed the task will actually run using celery UNLESS I daemonize the task using the --detach flag. I don't know why the task would fail only when run as a daemon. I have tried setting the same userid and groupid with the -u and -g flags, the same umask, and ensuring the path and working directories are the same for both the daemon and the non-daemonized version, but the task still will not run in celery when celery is running as a daemon.

I"m using the latest version of celery (0.9.4).

Answer 1

You're connecting to a port that is not speaking TLS. Are you trying to talk to a TLS/SSL enabled mail server, or is celery trying to use TLS for its AMQP connection?

Answer 2

If it only breaks when run inside the celery worker, there might be something with amqplib (which uses the ssl module) or it could be something with multiprocessing and forking (a global variable that was initialized before the fork that is no longer alive)

Could you please include the task you're trying to run? Do you connect to the server inside the task itself, or is it some kind of shared object?

Answer 3

According to the creator of celeryd:

Celery no longer does its own detaching as of 01a8a0e. There has been far too many problems with it, and since it works when detaching using start-stop-daemon, supervisord, launchd and so on, you are encouraged to use those tools instead.