As others have said, CruiseControl is a must. Also, make sure you get some NUnit for automated testing, and maybe WatiN if it fits to your needs. Also, NCover is good to have, but I think they charge for it now.
Also think about getting RedGate Ants for performance testing. It costs some money, but it saves you a lot of time in preempting performance issues, so it pays for itself very quickly.
There are a lot of recommendations for DVCS source control like Git and Mercurial, but by all means look at less-distributed source control options as well. Much like with Agile, some DVCS evangelists may tell you that it is the one true new correct way to do it, and that if you are using the old fashioned systems you are a dinosaur that will quickly go out of business. Of course, this is silliness, and traditional source control systems like SVN and Vault work just fine, especially if you are a small shop that is not going to have too much branching and forking. Personally, I prefer Vault, because it is more end-user friendly, it integrates into Visual Studio more cleanly, the inevitable branching and merging are easier, and it's ridiculously simple to setup. However, SVN is free while Vault costs about $300/user, but im my experience we've saved several multiples of that due to developer-time-saved with Vault. Regardless, just don't use SourceSafe or CVS.
If you already have the Visual Studio Team System tools, you may be tempted to use them, but I would recommend staying away from them unless you have a lot of time to kill orif you really want an ulcer. My company is a BizSpark member, so we get plenty of licenses for TFS and all of their ancillary tools (build, automated test, performance test, etc), and we don't use ANY of them. TFS is a massive timesink that requires an enormous amount of effort to setup and maintain (I've lost several days trying to get it working, just imagine what THAT costs), and the other tools fall short of the existing components they were intended to replace. Their build server is not as good as Cruise Control, their unit testing compoents are not as good as NUnit, their performance testing is not as good as Ants, etc.
For bug tracking, we use BugNet, mostly because it was free and .NET-based, but it's pretty limited and I'm sure there are plenty of other good ones out there. JIRA works pretty well too, and includes more project management functionality, but I think it costs a fair amount of money.
If you're ever going to prepare a screenshot or mockup for a proposal or design document, use Balsamiq. I forget what it costs, but it's great and worth it.
If you're going to write any blogs, WordPress is really easy to setup, free, and has a ton of professional-looking plugins and skins for free.