You need to differentiate between .NET versions and C# versions.
If you write code in VS 2010 (or otherwise compile with the C# 4 compiler) but only use C# 2 features and only target types in the .NET 2.0 framework, then yes, you'll be able to compile with the C# 2 compiler (in Visual Studio 2005) too. Your project file may or may not work, of course - but the raw source code should be okay.
Note that you can sometimes use new features without being aware that you're doing so - times where later versions of the language make previously-invalid conversions valid, for example, can change which method overload is picked or what method conversion is applied. These are corner cases, but they can happen. Note that telling csc to compile with an old language version doesn't always disable these conversions... there are cases where the C# 3 compiler will behave differently to the C# 4 compiler set to use version 3.