Why is msbuild and link.exe "hanging" during a build?

Question

We have a few C++ solutions and we run some build scripts using batch files that call msbuild.exe for each of the configurations in the solutions.

This had been working fine on 3 developer machines and one build machine, but then one of the projects started to hang when linking. This only happens on the newest machine which is a quad core, 2.8ghz I think. It runs on Windows Server 2003 and the others are on XP or Vista.

This happens consistently even if I change the order of builds in the bat file.

If I run the build from the IDE on that machine it does not hang.

Any ideas about what could possibly be causing this?

I am using Visual Studio 2008.

---

Edit:

I see now that when it is hung the following are running:

• link.exe (2 instances) One with large memory usage and one with a small amount of memory usage.
• vcbuild.exe
• msbuild.exe
• vcbuildhelper.exe
• mspdbsrv.exe

---

Edit:

The exe file exists and so does the pdb file.

The exe file is locked by some process, and I can't delete it or move it. I can delete the pdb file though.

I also have the problem if I just use VCBuild.exe.

I decided to try debugging the 2 link.exe processes and the mspdbsrv.exe processes.

When I attached the debugger/MSdev IDE to them I got a message box saying that the application was deadlocked and/or that "all threads have exited".

I guess I will have to check for a service pack for that msdev install on that machine.

---

Edit:

In the debug.htm output file I get all sorts of stuff output after the link.exe command is generated.

However, for the release buildlog.htm the linke.exe line is the last line.

This is clearly a hang in the linker. Definitely a Microsoft bug.

I am now trying to figure out what the .rsp (linker response) file is.

When I issue:

link.exe @c:\\Release\RSP00000535202392.rsp /NOLOGO /ERRORREPORT:QUEUE

That is the last line in the release build log. The debug one has lots more information after that.

Reinstalling a different version of Visual Studio did not solve the problem.

I will open an issue/ticket with Microsoft. I will post an answer if I can.

Answer 1

Are you using xcopy in your scripts? This suggests wrapping xcopy with cmd /c " .. " as a solution.

If that wasn't it, I'd recommend to narrow things down by only letting one cpu work (i.e. removing /maxcpucount) This would rule out any form of race condition between compilation processes.

Answer 2

Have you tried disabling incremental linking, or alternatively, always forcing a Rebuild All?

Answer 3

You could try this: Open the build dialog via Menu -> Tools -> Options -> Projects and Solutions -> Build and Run

Here you can set "MSBuild project build output verbosity" to "Diagnostic". Maybe this will deliver more information on what is going wrong.

In the same dialog you can set "Maximum number of parallel project builds" to 1. Maybe this works around the link.exe "hang".

Answer 4

Whole-program optimization (/GL and /LTCG) and /MP don't mix -- the linker hangs. I raised this on Connect.

The upshot is that it's a confirmed bug in VS2008; contact PSS if you want a hotfix; and the fix is included in VS2010.

If you can't wait that long, turn off /MP (slower compiles) or /LTCG (slower code).

Answer 5

mspdbsrv.exe is used to combine all debug info into one pdb file. The VS2005 version of mspdbsrv.exe is buggy, it might be that the VS2005 version has some of the same issues. Killing it before building is making a difference for some people. We're going to add it to our builds as well since we're regularly suffering from unknown PDB errors.