msbuild.exe staying open, locking files

Question

I use TeamCity which in turn invokes msbuild (.NET 4). I have a strange issue in that after a build is complete (and it doesn't seem to matter if it was a successful build or not), msbuild.exe stays open, and locks one of the files, which means every time TeamCity tries to clear its work directory, it fails, and can't continue.

This happens almost every time.

I'm really lost on this one, so I'll try to provide as much detail as possible.

• Server is an Intel Core i7, 2 GB ram, with Windows Server 2008 standard 64-bit SP2.
• In TeamCity, the msbuild runner is configured with the /m command-line parameter (which means to use multiple cores)
• The file in question is ALWAYS the same external DLL that is referenced in one of the .NET projects, in the path External Tools\Telerik\Telerik.Reporting.Dll. (There are several other .DLL files included in the External Tools dir in a similar path structure which never cause this problem). Currently this is with the trial version of Telerik reports, in case that makes any difference.
• When the issue happens, there are always several msbuild.exe *32 processes listed in Task manager: I believe there are 7. Using Process Explorer, they all look like top-level processes (no parents). They're all using from 20-50MB ram, and 0.0% CPU.
• If I wait 1-3 minutes, the msbuild.exe processes exit on their own, and TeamCity can then update the work directory properly.
• If I manually terminate the msbuild processes, TeamCity's update will work again immediately.
• Indexing services are turned off in Windows (though the prior two points pretty much confirm it's msbuild.exe causing the problem).
• There are no special properties on Telerik.reporting.dll. The only SVN property is svn:mime-type = application/octet-stream

Has anyone run across this before?

Answer 1

Use msbuild with /nr:false.

Briefly: msbuild tries to do lots of things to be fast, especially with parallel builds. It will spawn lots of "nodes" - individual msbuild.exe processes that can compile projects, and since processes take a little time to spin up, after the build is done, these processes hang around (by default, for 15 minutes, I think) so that if you happen to build again soon, these nodes can be "reused" and save the process setup cost. But you can disable that behavior by turning off nodeReuse with the aforementioned command-line option.

See also

http://connect.microsoft.com/VisualStudio/feedback/details/554791/msbuild-and-conhost-remain-in-memory-after-parallel-build

http://msdn.microsoft.com/en-us/library/ms164311.aspx

http://devnet.jetbrains.net/thread/286796

http://blogs.msdn.com/b/msbuild/archive/2007/04/16/node-reuse-in-multiproc-msbuild.aspx