Why do Test::Unit testcases start up so slowly?

Question

>rails -v
Rails 1.2.6

>ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32]

When I run a test fixture (that tests a rails model class) like this, it takes 20-30 secs to start executing these tests (show the "Loaded suite..."). What gives?

>ruby test\unit\category_test.rb

require File.dirname(__FILE__) + '/../test_helper'

class CategoryTest < Test::Unit::TestCase
    def setup
     Category.delete_all
    end
    def test_create
     obCategoryEntry = Category.new({:name=>'Apparel'})
     assert obCategoryEntry.save, obCategoryEntry.errors.full_messages.join(', ')
     assert_equal 1, Category.count

     assert_not_nil Category.find(:all, :conditions=>"name='Apparel'")
    end
    #.. 1 more test here
end

This one is Rails using a MySql DB with no fixtures. This time it clocked 30secs+ to startup.

Answer 1

It seems like Test::Unit is the simplest, but also one of the slowest ways to do unit testing with Ruby. One of alternatives is ZenTest.

Answer 2

Test unit startup isn't particularly slow, and nowhere near 20 seconds.

(11:39) ~/tmp $ cat test_unit.rb 
require 'test/unit'
class MyTest < Test::Unit::TestCase
  def test_test
    assert_equal("this", "that")
  end
end

(11:39) ~/tmp $ time ruby test_unit.rb 
Loaded suite test_unit
Started
F
Finished in 0.007338 seconds.

  1) Failure:
test_test(MyTest) [test_unit.rb:4]:
<"this"> expected but was
<"that">.

1 tests, 1 assertions, 1 failures, 0 errors

real    0m0.041s
user    0m0.027s
sys     0m0.012s

It's probably something you're doing in your tests. Are you doing anything complicated? Setting up a database? Retrieving something from the internet?

Answer 3

When starting any tests, Rails first loads any fixtures you have (in test/fixtures) and recreates the database with them.

20-30 seconds sounds very slow though. Do you have a lot of fixtures that need to be loaded before your tests run, or is your database running slow?

Answer 4

What does your test_helper.rb look like? Are you using instantiated fixtures?

self.use_instantiated_fixtures  = true

[edit]

If this is set to true try setting it to false.

Answer 5

Complete shot in the dark, but the majority of the time I see long startup times on things, it is usually due to some sort of reverse DNS lookup happening with some TCP socket communication somewhere along the way.

Try adding:

require 'socket'
Socket.do_not_reverse_lookup = true

at the top of your test file after your other require line.

Answer 6

Take a look at this Rails Test Server.

A quote from the author:

"Every time you run a test in a Rails application, the whole environment is loaded, including libraries that don’t change between two consecutive runs. That can take a considerable amount of time. What if we could load the environment once, and only reload the changing parts before each run? Introducing RailsTestServing.

With RailsTestServing, the run time of a single test file has gone from 8 seconds down to .2 of a second on my computer. That’s a x40 speed improvement. Now, I don’t think twice before hitting ⌘R in TextMate. It feels liberating!"

(This was featured on the Rails Envy Podcast this past week which is where I found this.)

Answer 7

Ruby's gem tool follows a path discovery algorithm which, apparently, is not Windows (as I see from your ruby -v) friendly.

You can get a clear picture if you trace, for example, a Rails application loading with ProcMon. Every (I really mean every) require starts a scan over all directories in Ruby's path plus all gem directories. A typical require takes 20 ms on an average machine. Since Rails makes hundreds of requires, those 20 ms easily sum up to seconds every time you launch the Rails environment. Take in the time to initialize the fixtures in the database and you get a better idea of why it takes so much time to just begin running the test-cases.

Perhaps because of each file-system architecture and implementation (path caching etc.), this is less of a problem in Linux than in Windows. I don't know who you should blame, though. It looks like the NTFS file-system could be improved with a better path caching implementation, but clearly the gem tool could implement the caching itself and have its performance not so dependent on the platform.