Hudson failing build w/o revealing cause

Question

Every build has failed as of Tuesday. I'm not exactly sure what happened. The Phing targets (clean/prepare) are being executed properly. Additionally, the unit tests are passing with flying colors, with only a warning for duplicate code (not a reason for a fail). I tried removing the phpDoc target to see if that was causing the error, but the build still failed.

Started by user chris Updating file://localhost/projects/svn/ips-com/trunk At revision 234 no change for file://localhost/projects/svn/ips-com/trunk since the previous build [trunk] $ /opt/phing/bin/phing clean prepare -logger phing.listener.NoBannerLogger Buildfile: /var/lib/hudson/.hudson/jobs/IPS/workspace/trunk/build.xml

IPS > clean:

 [echo] Clean...    [delete] Deleting directory

/var/lib/hudson/.hudson/jobs/IPS/workspace/build

IPS > prepare:

 [echo] Prepare...
[mkdir] Created dir: /var/lib/hudson/.hudson/jobs/IPS/workspace/build
[mkdir] Created dir: /var/lib/hudson/.hudson/jobs/IPS/workspace/build/logs
[mkdir] Created dir: /var/lib/hudson/.hudson/jobs/IPS/workspace/build/logs/coverage
[mkdir] Created dir: /var/lib/hudson/.hudson/jobs/IPS/workspace/build/logs/coverage-html
[mkdir] Created dir: /var/lib/hudson/.hudson/jobs/IPS/workspace/build/docs
[mkdir] Created dir: /var/lib/hudson/.hudson/jobs/IPS/workspace/build/app

BUILD FINISHED

Total time: 1.0244 second

[workspace] $ /bin/bash -xe /tmp/hudson3259012225710915845.sh + cd trunk/tests + /usr/local/bin/phpunit --verbose -d memory_limit=512M --log-junit ../../build/logs/phpunit.xml --coverage-clover ../../build/logs/coverage/clover.xml --coverage-html ../../build/logs/coverage-html/ PHPUnit 3.5.0 by Sebastian Bergmann.

IPS Default_IndexControllerTest .

Default_AuthControllerTest ......

Manage_UsersControllerTest .....
testDeleteInvalidUserId .. testGetPermissionsForInvalidUserId .. Audit_OverviewControllerTest ............

Time: 14 seconds, Memory: 61.00Mb

[30;42m[2KOK (28 tests, 198 assertions) [0m[2K Writing code coverage data to XML file, this may take a moment.

Generating code coverage report, this may take a moment.

Warning: Unknown: Error occured while closing statement in Unknown on line 0

Warning: Unknown: Error occured while closing statement in Unknown on line 0

Warning: Unknown: Error occured while closing statement in Unknown on line 0

Warning: Unknown: Error occured while closing statement in Unknown on line 0

Warning: Unknown: Error occured while closing statement in Unknown on line 0

Warning: Unknown: Error occured while closing statement in Unknown on line 0

Warning: Unknown: Error occured while closing statement in Unknown on line 0

Warning: Unknown: Error occured while closing statement in Unknown on line 0 [workspace] $ /bin/bash -xe /tmp/hudson1439023061736436000.sh + /usr/local/bin/phpcpd --log-pmd ./build/logs/cpd.xml ./trunk phpcpd 1.3.2 by Sebastian Bergmann.

Found 1 exact clones with 6 duplicated lines in 2 files:

• library/Ips/Form/Decorator/SplitInput.php:8-14 library/Ips/Form/Decorator/FeetInches.php:10-16

0.04% duplicated lines out of 16585 total lines of code.

Time: 4 seconds, Memory: 19.50Mb [DRY] Skipping publisher since build result is FAILURE Publishing Javadoc [xUnit] [INFO] - Starting to record. [xUnit] [WARNING] - Can't create the path /var/lib/hudson/.hudson/jobs/IPS/workspace/generatedJUnitFiles. Maybe the directory already exists. [xUnit] [INFO] - Processing PHPUnit-3.4 (default) [xUnit] [INFO] - [PHPUnit-3.4 (default)] - 1 test report file(s) were found with the pattern 'build/logs/phpunit.xml' relative to '/var/lib/hudson/.hudson/jobs/IPS/workspace' for the testing framework 'PHPUnit-3.4 (default)'. [xUnit] [INFO] - Converting '/var/lib/hudson/.hudson/jobs/IPS/workspace/build/logs/phpunit.xml' . [xUnit] [INFO] - Stopping recording. Publishing Clover coverage report... Publishing Clover XML report... Publishing Clover coverage results... Finished: FAILURE

Answer 1

What changed since Tuesday? Try to manually run exactly the same commands that Hudson tries to run from the same directory that Hudson starts it from (usually the jobs workspace directory). Of course with the user account that Hudson is started under.

There are several possibilities. ranging from standard groups for a directory, to permission, or other things outside of Hudson. Was Hudson upgraded? Was a plugin upgraded? Was the OS or php upgraded? Was there a change in the default or user .profile or .env (or the equivalent files)? Does another process accesses the workspace? ......

Once I had the problem that all of the sudden my deployment scripts did not run anymore. The mystery was, that I could still run the script from command line with the Hudson user account. The reason was simple but took a while to uncover. There was a java upgrade from 5 to 6. Both versions were available. After comparing the environment variables, there was a difference in the path. The problem was that the new path was set in the global .profile. But Hudson does not open an interactive shell, therefore the .profile will not be executed. If you have a problem like this, you can put the initialization in the .env file (or whatever the filename is for your system), because this will be run regardless if it is a interactive shell or not. Alternatively you can configure Hudson to set it on master or node/slave level.