Here is what it was. My application has two forms - login and main form where all the action happens. The login form has two buttons (Login and Cancel). Login button logs user in, closes the login form and opens the main form. Cancel button just closes the login form. To close the form I simply used this.Close().
What was happening though is that I still needed to dispose of the login form explicitly by doing something like:
frmLogin.Dispose();
frmLogin = null;
before exiting the program (in my Program.cs)
So this solved it. I had to make sure that this was being done in both cases: when the user logs in as well as when they choose to not log in.
Crucial fact is that frmLogin is modal, hence Dispose() is not called automatically when closed.