It's possible to do this, but you'll need to mess with the project file manually.
We do this in MiscUtil so we can have a .NET 2.0 build and a .NET 3.5 build. For instance:
<ItemGroup Condition=" '$(Configuration)' != 'Release 2.0' ">
<Reference Include="System.Core">
<RequiredTargetFramework>3.5</RequiredTargetFramework>
<Aliases>global</Aliases>
</Reference>
<Reference Include="System.Xml.Linq">
<RequiredTargetFramework>3.5</RequiredTargetFramework>
</Reference>
</ItemGroup>
That should be enough to get you started :) Basically take the current reference out of where it is in your normal project file, and put it in its own ItemGroup with an appropriate condition.