There are several ways to do this, all of which have their advantages and disadvantages, and googling will give you a lot of discussion about it.
The methods you describe have the weaknesses you mention. Also you need to consider how unique you want your instance to be:
per WindowStation
per machine
I have a class that can be used to force a single instance per WindowStation for Windows Forms applications. Basically it uses P/Invoke to do the following during startup, protecting against a race condition using a Mutex:
Define a fixed unique string to identify your app (e.g. a GUID)
Obtain a Mutex whose name is derived from your unique string.
Call EnumWindows using P/Invoke to enumerate all main windows.
In the EnumWindows callback, call GetProp using P/Invoke on each window in turn, passing the window handle and your unique value as arguments. If GetProp detects a window that has your unique key (i.e. another instance of your application), activate it (PostMessage(...SC_RESTORE...), SetForegroundWindow, SetFocus), then exit your new instance.
If you enumerate all windows without finding a previous instance, call SetProp using P/Invoke to set your unique value for your application's main form window handle.
Release the mutex and continue starting up.
In my use case, this solution is better than simply holding a mutex for the lifetime of the process, because it gives a way to activate the previous instance if the user starts a second instance rather than just exiting. Using GetProp/SetProp gives you a way of identifying your process instance without relying on the process name.