To answer your first question, in Windows Server 2003 or 2008, there is a thing called Windows System Resource Manager, aka WSRM, that lets you allocate system resources to apps or processes. This comes with Enterprise Edition of the Server OS. Just how to allocate resources is sort of a black art; you will have to try things out, test thoroughly in all sorts of conditions.
For your second question - best way to handle "heavy operations" - I guess the easy answer there is it depends. There are many different approaches, and they fit different requirements.
MSMQ can be helpful for buffering work requests to enable processing on off-peak hours. WSRM can enable allocating and rationing of system resources. These two can be complementary, or you may view them as alternatives.
As for ASP.NET and Windows Services: ASP.NET is obviously going to be primarily useful for browser-based requests, but also possibly for REST-style requests (or similar) from other systems. ASP.NET apps will run under the w3wp.exe worker process, which is nanny'd by IIS. You get all the IIS process management goodies like, periodic restart, auto restart of stale processes, request-based activation, and etc.
Windows Services is a way to create custom apps that run on your Windows Server, that may not be request-based. One example for your scenario might be a Windows Service that starts up on a schedule, let's say at 10pm every evening, and runs through all the queued-up transactions sitting in MSMQ. When the queue is empty or at 4am, whichever comes first, the Windows Service shuts down. A Windows Service does not enjoy any of the IIS process management benefits.
Bottom line, ASP.NET and Windows Services offer different alternatives for process or application hosting and lifecycle management.
This is all just general information. To get more specific and concrete guidance from this community, you will have to make a more specific request than "how can I do heavy processing?"