You'll probably need to manually edit the underlying .csproj file. Approximately thusly
<Reference Condition=" '$Configuration'=='Debug' "
Include="path\to\Debug\Foo.dll" />
<Reference Condition=" '$Configuration'=='Release' "
Include="path\to\Release\Foo.dll" />
(May be easy to add a reference to debug version via browse, then right click project in solution explorer, click 'Unload project', then right click again, 'Edit your.csproj', make the edit suggested above to the Foo.dll that you just added, then right click, 'Reload project'.)
EDIT
To prevent seeing two copies inside VS, maybe something like
<Reference debugstuff as before>
<Visible Condition=debugcond>true</Visible>
<Visible Condition=releasecond>false</Visible>
</Reference>
and same for release
That is, conditionally set Visible
metadata under the reference node to true/false based on the condition. I haven't tried to to know if it works.