Task Manager shows you the amount of memory belonging to the application that happens to be paged into real memory at the time. When you minimise an application in Windows, the kernel automatically trims the process working set so it takes less memory. This behaviour is not specific to .NET apps.
If you look instead at the process "Private Bytes" value, this will give you a better indication of the amount of memory used by a process. This value will not change when the process working set is trimmed.
Microsoft offers more information at this knowledgebase article: http://support.microsoft.com/kb/293215