Thinking Sphinx not working in test mode

Question

I'm trying to get Thinking Sphinx to work in test mode in Rails. Basically this:

ThinkingSphinx::Test.init
ThinkingSphinx::Test.start

freezes and never comes back.

My test and devel configuration is the same for test and devel:

dry_setting: &dry_setting
  adapter: mysql
  host: localhost
  encoding: utf8
  username: rails
  password: blahblah

development:
  <<: *dry_setting
  database: proj_devel
  socket: /tmp/mysql.sock  # sphinx requires it

test:
  <<: *dry_setting
  database: proj_test
  socket: /tmp/mysql.sock  # sphinx requires it

and sphinx.yml

development:
  enable_star: 1
  min_infix_len: 2
  bin_path: /opt/local/bin

test:
  enable_star: 1
  min_infix_len: 2
  bin_path: /opt/local/bin

production:
  enable_star: 1
  min_infix_len: 2

The generated config files, config/development.sphinx.conf and config/test.sphinx.conf only differ in database names, directories and similar things; nothing functional.

Generating the index for devel goes without an issue

$ rake ts:in
(in /Users/pupeno/proj)
default config
Generating Configuration to /Users/pupeno/proj/config/development.sphinx.conf
Sphinx 0.9.8.1-release (r1533)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file '/Users/pupeno/proj/config/development.sphinx.conf'...
indexing index 'user_core'...
collected 7 docs, 0.0 MB
collected 0 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 0.0 Mhits, 99.8% done
total 7 docs, 422 bytes
total 0.098 sec, 4320.80 bytes/sec, 71.67 docs/sec
indexing index 'user_delta'...
collected 0 docs, 0.0 MB
collected 0 attr values
sorted 0.0 Mvalues, nan% done
total 0 docs, 0 bytes
total 0.010 sec, 0.00 bytes/sec, 0.00 docs/sec
distributed index 'user' can not be directly indexed; skipping.

but when I try to do it for test it freezes:

$ RAILS_ENV=test rake ts:in
(in /Users/pupeno/proj)
DEPRECATION WARNING: require "activeresource" is deprecated and will be removed in Rails 3. Use require "active_resource" instead.. (called from /Users/pupeno/.rvm/gems/ruby-1.8.7-p249/gems/activeresource-2.3.5/lib/activeresource.rb:2)
default config
Generating Configuration to /Users/pupeno/proj/config/test.sphinx.conf
Sphinx 0.9.8.1-release (r1533)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file '/Users/pupeno/proj/config/test.sphinx.conf'...
indexing index 'user_core'...

It's been there for more than 10 minutes, the user table has 4 records.

The database directory look quite diferently, but I don't know what to make of it:

$ ls -l db/sphinx/development/
total 96
-rw-r--r--  1 pupeno  staff   196 Mar 11 18:10 user_core.spa
-rw-r--r--  1 pupeno  staff  4982 Mar 11 18:10 user_core.spd
-rw-r--r--  1 pupeno  staff   417 Mar 11 18:10 user_core.sph
-rw-r--r--  1 pupeno  staff  3067 Mar 11 18:10 user_core.spi
-rw-r--r--  1 pupeno  staff    84 Mar 11 18:10 user_core.spm
-rw-r--r--  1 pupeno  staff  6832 Mar 11 18:10 user_core.spp
-rw-r--r--  1 pupeno  staff     0 Mar 11 18:10 user_delta.spa
-rw-r--r--  1 pupeno  staff     1 Mar 11 18:10 user_delta.spd
-rw-r--r--  1 pupeno  staff   417 Mar 11 18:10 user_delta.sph
-rw-r--r--  1 pupeno  staff     1 Mar 11 18:10 user_delta.spi
-rw-r--r--  1 pupeno  staff     0 Mar 11 18:10 user_delta.spm
-rw-r--r--  1 pupeno  staff     1 Mar 11 18:10 user_delta.spp

$ ls -l db/sphinx/test/       
total 0
-rw-r--r--  1 pupeno  staff  0 Mar 11 18:11 user_core.spl
-rw-r--r--  1 pupeno  staff  0 Mar 11 18:11 user_core.tmp0
-rw-r--r--  1 pupeno  staff  0 Mar 11 18:11 user_core.tmp1
-rw-r--r--  1 pupeno  staff  0 Mar 11 18:11 user_core.tmp2
-rw-r--r--  1 pupeno  staff  0 Mar 11 18:11 user_core.tmp7

Nothing gets added to a log when this happens. Any ideas where to go from here?

I can run the command line manually:

/opt/local/bin/indexer --config config/test.sphinx.conf --all

which generates the output as the rake ts:in, so no help there.

Answer 1

The problem was the random ids generated by fixtures. The solution is described on http://freelancing-god.github.com/ts/en/common_issues.html#slow_indexing

Slow Indexing

If Sphinx is taking a while to process all your records, there are a few common reasons for this happening. Firstly, make sure you have database indexes on any foreign key columns and any columns you filter or sort by.

Secondly – are you using fixtures? Rails’ fixtures have randomly generated IDs, which are usually extremely large integers, and Sphinx isn’t set up to process disparate IDs efficiently by default. To get around this, you’ll need to set sql_range_step in your config/sphinx.yml file for the appropriate environments:

development:
  sql_range_step: 10000000

I added it to both, development and test environments.