I'm doing some largish scale data processing with nltk. In doing so we iterate through 2-3000 files sitting on the server, reading through them all, making changes, and closing them out. This is all handled via Celery tasks. The only problem is that we often find ourselves encountering this:
IOError: [Errno 5] Input/output error
A quick restart of Celeryd, and we're instantly back in business. My problem is that this is seemingly happening daily. What could be causing it? Any ideas on how to track it down? It occurs on any print/logging statement, basically anytime we need to input/output anything whatsoever. Once Celery is in this state, any/all future jobs submitted will do the exact same thing. Anyone ran into similar stuff? Perhaps have a clue as to why this would be occurring?
EDIT: Celery version 1.0.5 -- The next version up I believe was a pretty large shift, so we haven't had time to upgrade quite yet.