I am maintaining a .net 3.5sp1 website. I'd like to upgrade a part of the site to 4.0 so I can start converting to SilverLight 4.0 & RIA services.
Can I create a subdomain of my main site, and have the subdomain be running 4.0 while the main website runs 3.5?
4.0 uses a new runtime. That means the two sections of your site are in different app domains, and that means certain things like Session and authentication tokens won't carry over well. It doesn't mean you can't do it, but there will be some challenges to overcome.
If you mean subdomain as in:
http://aaaaa.yourdomain.com and http://bbbbb.yourdomain.com
then yes, there's no reason at all why not.
Assuming you're using Windows Server 2003 or later, and IIS, host headers let you run multiple web sites at a single IP address, and you can specify the version of the runtime to be used for each one independently.
I run Windows Server 2003 with about 20 separate sites on it, all subdomains of a single domain, and they're a mixture of ASP.NET 1.1 and 2.0.
You specify the runtime version for each site in IIS manager, under Web site > Properties > ASP.NET version. Not sure if it's exactly the same in Windows Server 2008, but it can't be all that different.