Can you do parallel incremental building on a list of projects?

Question

Is it possible to take a list of projects and parallel build them only if they aren't up to date?

<Target Name="DependenciesLevel7" DependsOnTargets="DependenciesLevel6">
<Message Text="5 items to build" />
<MSBuild Projects="C:\Projects\ApplicationManager.csproj;C:\Projects\Metrics.csproj" Properties="$(CustomAllProperties)" BuildInParallel="true">
  <Output TaskParameter="TargetOutputs" ItemName="built_DependenciesLevel7" />
</MSBuild>

This is an example of the format i'm building in, I was hoping to be able to parallel build only items that aren't up to date here? Perhaps internal msbuild task calls are automatically parallel? If so how would I set this up so that it's incremental based on the previous build task? (Target DependenciesLevel6) I believed for incremental building you have to use Inputs/Outputs in your target.

Question summary:

• Is the list of projects passed to an msbuild task automatically incremental (building is skipped if the build is already up to date)?
  • If not, is it possible to do incremental on a list of projects while parallelizing?
• Can you do between target incremental where each target is a parallel build?

Answer 1

You're probably looking for IncrediBuild.

Answer 2

I'm not sure about the incremental part but I read that it checks inputs/ouputs.

I make parallel builds with MsBuild with a configuration similar to yours, don't forget to run msbuild with the /m option.

You can try on a simple project and check the logs to see if it's incremental or not.

Answer 3

Hi, what you are describing "parallel incremental building" is already built in (kind of) but what you really need is parallel partially building.

Let me first explain the concepts and then I'll circle back how it works in your scenario specifically. There are two things you need to know about; incremental building and partial building. I just wrote a blog post discussing this at http://sedodream.com/2010/09/23/MSBuildYouveHeardOfIncrementalBuildingButHaveYouHeardOfPartialBuilding.aspx but I'll paste the relevant parts here.

Incremental building is the concept that you should only build what is out of date. To support this MSBuild has the attributes, inputs and outputs on the Target element. With these attributes you can specify the files that go into a target (via inputs attribute), and the files that you are expecting to come out of a target (via outputs attribute). Once you do this MSBuild will compare the timestamp of the inputs to the outputs and if all outputs are up-to-date (i.e. the inputs are older) then the target will be skipped. Take a look at the very simple project file below.

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"&gt;

  <ItemGroup>
    <Files Include="src\01.txt;src\02.txt;src\03.txt;src\04.txt;src\05.txt;"/>
  </ItemGroup>

  <PropertyGroup>
    <Dest>dest\</Dest>
  </PropertyGroup>

  <Target Name="CopyFiles" 
          Inputs="@(Files)" 
          Outputs="@(Files->'$(Dest)%(Filename)%(Extension)')">

    <Message Text="CopyFiles" />
    <Copy SourceFiles="@(Files)"
          DestinationFiles="@(Files->'$(Dest)%(Filename)%(Extension)')"/>
  </Target>

  <Target Name="DeleteTwoFiles">
    <Message Text="DeleteTwoFiles" />
    <Delete Files="$(dest)01.txt;$(dest)02.txt"/>
  </Target>
</Project>

In this project file we have two targets; CopyFiles and DeleteTwoFiles. Ignore DeleteTwoFiles for now. Also take a note that the directory where I’m executing this build has a folder, src, with the files listed in the Files item. On the CopyFiles target I have specified the inputs and outputs. The inputs is just @(Files), this are the files that the target is acting upon. The outputs contains the expression @(Files->'$(Dest)%(Filename)%(Extension)'). Which is the same expression from the Copy statement. If the Dest folder is empty and I execute the CopyFiles target the result is shown below.

So just as expected the files were copied over, so its all good. Now what happens if I execute it again? The output is shown below

So as you can see the target was skipped, the message statement “CopyFiles” was not executed nor was the copy as a result. So this, in a nutshell, is incremental building. Now, with the dest folder containing all files, what do you think would happen I execute the command msbuild.exe PartialBuilding01.proj /t:DeleteTwoFiles;CopyFiles? This command will first delete two files from the output directory and then call the CopyFiles target again. Let’s see the result below.

When the CopyFiles target was executed you see that statement “Building target ‘CopyFiles’ partially, …”. When the time came to execute the target MSBuild examined the inputs and outputs, it determined that the files 01.txt & 02.txt were out of date (because they didn’t exist in the target) but 03.txt, 04.txt and 05.txt were up to date. So MSBuild feed the CopyFiles target a value for the Files item that only contained the 01.txt and 02.txt and let it do its thing.

Now this relates to your problem in many ways some not as direct as you might hope. Firstly MSBuild will incrementally build your project, so if your project is up to date then it will not be built again. The thing is though that in order for MSBuild to determine that your project is up to date it has to load the project run the default target (usually Build) and then the targets themselves will figure out that there is no work to do. This stuff itself takes time. So if you have a huge number of projects, or a huge number of files inside of a project then you can take matters into your own hands. What you need is a way to determine if your projects are up to date or not and correctly express that inside of your inputs and outputs attributes. Once you do this you should be able to skip building the projects which are up to date.

The core of the problem is how do you craft the inputs/outputs to be correct. If you can think of a way to do that then you will get what you want. How you craft this will depend on your scenario but I could see something like this:

1. After each project build drop a file to a known location that is specific to that project
2. Before you build a project scan its directory, find the newest file and then update the timestamp of the project file to be that value
3. Then you can place the project files as the Inputs values and the marker files as the Outputs
4. Then call your target

In this case you assume that all dependencies are fully contained in files under the directory of the project (which may not be true). I'm not saying this is the ideal solution, but a solution.

==============================================

Edit: Update based on questoins below.

You will want to put the projects into an item (though not required) like ProjectFiles and then use @(ProjectFiles) for inputs. For outputs that is what I was saying is the hard part. You have to figure out a way to know (or indicate to you via your own process) that the projects are up to date. There is nothing built in for this. Concern fo incremental build vs. clean build. In a perfect world incremental & clean builds are the same. But sometimes that is not the case. For many projects it is. If you start adding a bunch of targets to your build process and you set them up to do incremental build, but you do not implement that properly then you may MSBuild may skip targets when they were indeed out of date. A good example of this would be when you create a target with Inputs set to a list of files and then the Outputs set to a list of created files. If you do not extend the clean process to delete those created files, then the next time you Rebuild (assuming you didn't change the files) the target will be skipped when it should have been cleaned on the previous Rebuild.