In addition to all the previous good comments, please remember that the numbers you are seeing in Task Manager are virtual memory, not physical memory. The actual amount of physical RAM used by the program is not obvious from looking at Task Manager. Also, Task Manager is showing you a series of snapshots - if you really had a problem, you'd want to look at it over time, with Perfmon or something, in addition to the excellent suggestion on profiling with JetBrains' dotTrace.
But don't do any optimization until your code works, and has high test coverage, and until you know what actually needs to be optimized. Otherwise, you run the risk of optimizing problems that don't exist, and worse, of optimizing the wrong problem, while ignoring the real problem or possibly making it worse.