OK, I found a big hint here from Anders Norås that there is a constructor for the CSharpCodeProvider constructor that takes some options, including the compiler version. When I checked the MSDN docs, I found that it's cleaner to specify the compiler options in the App.config file. Here's an example:
<system.codedom>
<compilers>
<compiler
language="vb;vbs;visualbasic;vbscript"
extension=".vb"
type="Microsoft.VisualBasic.VBCodeProvider, System,
Version=2.0.3600.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089"
compilerOptions="/optimize"
warningLevel="1" >
<providerOption
name="CompilerVersion"
value="v3.5" />
</compiler>
<compiler
language="c#;cs;csharp"
extension=".cs"
type="Microsoft.CSharp.CSharpCodeProvider, System,
Version=2.0.3600.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089"
compilerOptions="/optimize"
warningLevel="1" >
<providerOption
name="CompilerVersion"
value="v3.5" />
</compiler>
</compilers>
</system.codedom>
My only disappointment is that now I get two versions of each compiler from CodeDomProvider.GetAllCompilerInfo(). The configuration documentation says that the App.config should override the machine.config settings, but I get both. So far, the App.config is always after the machine.config, so I just let the later ones override the earlier ones. Anybody know how to avoid the duplication?