What steps should you take to speed up SimpleTest?

Question

I'm writing some testing code on a Drupal 6 project, and I can't believe how slow these tests seem to be running, after working with other languages and frameworks like Ruby on Rails or Django.

Drupal.org thinks this question is spam, and won't give me a way to prove I'm human, so I figured SO is the next base place to ask a question like this, and get a sanity check on my approach to testing.

The following test code in this gist is relatively trivial.

http://gist.github.com/498656

In short I am:

• creating a couple of content types,
• create some roles,
• creating users,
• creating content as the users,
• checking if the content can be edited by them
• checking if it's visible to anonymous users

And here's the output when I run these tests from the command line:

Drupal test run
---------------

Tests to be run:
 -  (ClientProjectTestCase)

Test run started: Thu, 29/07/2010 - 19:29

Test summary:
-------------

ClientProject feature 52 passes, 0 fails, and 0 exceptions

Test run duration: 2 min 9 sec

I'm trying to run tests like this before I push code to a central repo everytime, but if it's taking this long this early on the project, I dread to think about it further down the line when we have ever more test cases.

What can I do to speed this up?

I'm using a MacbookPro with:

• 4gb of ram,
• 2.2ghz Core 2 Duo processor,
• PHP 5.2,
• Apache 2.2.14, without any opcode caching, Mysql 5.1.42 (Innodb tables are my default)
• A 5400 RPM laptop hard drive

I understand that in the examples above I'm bootstrapping Drupal each time, and this is a very expensive operation, but this isn't unheard with other frameworks, like Ruby on Rails, or Django, and I don't understand why it's averaging out at a little over a minute per testcase on this project.

There's a decent list of tricks here for speeding up Drupal 7, many of which look like they'd apply to Drupal 6 as well, but I haven't yet had a chance to try them yet, and it would be great to hear how these have worked out for others by I blunder down further blind alleys,

What has worked for you when you've been working with Drupal 6 in this situation, and where are the quick wins for this?

One minute per test case when I'm expecting to easily more than a hundred test cases feels insane.

Answer 1

Google? Copypaste the relevant part of your question: http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=speed+up+SimpleTest+when+developing+with+Drupal%3F

the first is http://drupal.org/node/466972

is this hard?

Answer 2

It looks like the biggest increase in speed will come from running the test database in a ram disk, based on this post here on Performance tuning tips for Drupal 7 testing on qa.drupal.org

DamZ wrote a modified mysql init.d script for /etc/init.d/mysql on Debian 5 that runs MySQL databases entirely out of tmpfs. It's at http://drupal.org/files/mysql-tmpfs.txt, attached to http://drupal.org/node/466972.

It allowed the dual quad core machine donated to move from a 50 minute test and huge disk I/O with InnoDB to somewhere under 3 minutes per test. It's live as #32 on PIFR v1 for testing.d.o right now. It is certainly the only way to go.

I have not and won't be trying it on InnoDB anytime soon if anyone wants to omit the step on skip-innodb below and try it on tmpfs.

Also there some instructions here for creating a ram disk on OS X, although this is for moving your entire stock of mysql databases into a ram disk, instead of just a single database:

Update - I've tried this approach now with OS X, and documented what I've found

I've been able to cut 30-50% from the test times by switching to a ram disk. Here are the steps I've taken:

Create a ram disk

I've chosen a gigabyte mainly because I've got 4gb of RAM, and I'm not sure how much space I might need, so I'm playing it safe:

    diskutil erasevolume HFS+ "ramdisk" `hdiutil attach -nomount ram://2048000`

Setup mysql

Next I ran the mysql install script to get mysql installed on the new ramdisk

    /usr/local/mysql/scripts/mysql_install_db \
        --basedir=/usr/local/mysql \
        --datadir=/Volumes/ramdisk

Then, I took the following steps: I made sure the previous mysqld was no longer running, and then started the mysql daemon, making sure we tell it to use ram disk as our data directory, rather than the default location.

  /usr/local/mysql/bin/mysqld \
      --basedir=/usr/local/mysql \
      --datadir=/Volumes/ramdisk \
      --log-error=/Volumes/ramdisk/mysql.ramdisk.err \
      --pid-file=/Volumes/ramdisk/mysql.ramdisk.pid \
      --port=3306 \
      --socket=/tmp/mysql_ram.sock

Add the database for testing

I then pulled down the latest database dump on our staging site with drush, before updating where settings.php points to it:

drush sql-dump > staging.project.database.dump.sql

Next was to get this data into the local testing setup on the ram disk. This involved creating a symlink to the ramdisk database socket, and creating the database, granting rights to the mysql user specified in the drupal installation, then loading the database in to start running tests. Step by step:

Creating the symlink - this because the mysql command by default looks for /tmp/mysql.sock, and symlinking that to our short term ram disk was simpler than constantly changing php.ini files

ln -s /tmp/mysql_ram.sock /tmp/mysql.sock

Creating the database (from the comamnd line at the mysql prompt)

CREATE DATABASE project_name;
GRANT ALL PRIVILEGES ON project_name.* to db_user@localhost IDENTIFIED BY 'db_password';

Loading the content into the new database...

mysql project_database < staging.project.database.dump.sql  

Run the tests on the command line

...and finally running the test from the command line, and using growlnotify to tell me when tests have finished

php ./scripts/run-tests.sh --verbose --class ClientFeatureTestCase testFeaturesCreateNewsItem ; growlnotify -w -m "Tests have finished."

Two test cases takes around a minute and half still, is still unusably slow - orders of magnitude slower than other frameworks I might have used before.

What am I doing wrong here?

This can't be the standard way of running tests with Drupal, but I haven't been able to find any stats on how long I should expect a test suite to take with Drupal, to tell me otherwise,