Thinking Sphinx delta indexing fails in production

Question

Here's what I've determined:

1. Delta indexing works fine in development
2. Delta indexing does not work when I push to the production server, and no action is logged in searchd.log
3. I'm running Phusion Passenger, and, as recommended in the basic troubleshooting guide, have confirmed that:
   1. www-data has permission to run indexing rake tasks (ran them from command line manually)
   2. the path to indexer and searchd are correct (/usr/local/bin)
   3. there are no errors in production.log

What on earth could I possibly be missing? I'm running Ruby Enterprise 1.8.6, Rails 2.3.4, Sphinx 0.9.8.1, and Thinking Sphinx 1.2.11.

Thanks!

Answer 1

Are there any clues in Apache/Nginx's error log?

Answer 2

Here's the next troubleshooting step I would take. Open up the file for the delta indexing strategy you are using (presumably lib/thinking_sphinx/deltas/default_delta.rb). Find the line where it actually generates the indexing command. In mine (v1.1.6) it's line 20:

output = `#{config.bin_path}indexer --config #{config.config_file} #{rotate} #{delta_index_name model}`

Change that so you can log the command itself, and maybe log the output as well:

command = `#{config.bin_path}indexer --config #{config.config_file} #{rotate} #{delta_index_name model}`
RAILS_DEFAULT_LOGGER.info(command)
output = `#{command}`
RAILS_DEFAULT_LOGGER.info(output)

Deploy that to production and tail the log while modifying a delta-indexed model. Hopefully that will actually show you the problem. Of course maybe the problem is elsewhere in the code and you won't even get to this point, but this is where I would start.

Answer 3

Last night as I slept it hit me. Unsurprisingly, it was a stupid issue involving bad configuration, though I am rather surprised that it produced the results it did. I guess I don't know much about Thinking Sphinx internals.

Recently I migrated servers. sphinx.yml looked like this:

production:
  bin_path: '/usr/local/bin'
  host: mysql.mysite.com

On the new server, MySQL was just a local service, but I had forgotten to remove that line. Interestingly, manual rake reindexing still worked just fine. I'm intrigued that Thinking Sphinx didn't throw an error when trying to reload the deltas, since mysql.mysite.com no longer exists, even though that was clearly the source of the issue.

Thanks for all your help, and sorry to have brought up such a silly problem.

Answer 4

I was having this problem and found the "bin_path" solution mentioned above. When it didn't seem to work, it took me a while to realize that I'd pasted in the example code for "production" when I was testing on "stagiing" environment. Problem solved!

This was after making sure that the rake tasks that config, index, and start sphinx are all running as the same user as your passenger instance. If you log into the server as root to run these tasks, they will work in the console but not via passenger.

Answer 5

I had the same problem. Works on the command line, not inthe app.

Turns out that we still had a slave database that we were using for the indexing, but the slave wasn't getting updated.