As Roddy suggests, a parameter switch is I think the only reliable solution. Note however that if a user creates their own shortcut(s) this may not suit your needs.
i.e. if the user creates a shortcut that includes some "document" in the command line, do you want that to be treated as a command line invocation or a "double click". My guess is it's the latter.
If it's the former then you can stop reading here. :)
But if it is the latter then you may need to put a little more work in to (more) reliably get the behaviour you want....
That would involve using a command line switch to indicate command line launch, rather than a double click launch, since there are multiple ways to "effectively" double click, but only one way to run something from the command line - that is, USE the command line! (I consider Start Menu -> Run... to be a command line launch, by the way. ymmv)
So, if it were me I would create a command line specific launcher for the application. The command line launcher would actually be a very simple bit of code that takes the command line it is given and simply appends an additional parameter, before calling the actual application executable:
e.g.
myapp param1 param2
which simply calls ShellExecute with effectively:
myappgui param1 param2 /cmd
I myself would not be too concerned with having two executables since the user is far less concerned with the actual name of the exe representing your "real" app when invoking it thru the GUI because of course the GUI invocations of that don't involve knowing, or needing to know, the name. Your file-type associations and shortcuts etc simply all point directly to the "real" myappgui.exe, rather than myapp.exe
And your command line reference simply directs the user to use myapp.exe to invoke your app from a command line.
In the application, if it finds the command line switch then it knows it was launched via the command line launcher application, otherwise it must have been a double-click/GUI launch.
As far as the majority of your users are concerned, they may never even need to know that there are two exe's involved. In any event, you have to accept that if/when a user discovers your technique - which ever you choose - they can spoof it by fabricating a command line to achieve whatever ends they desire.
If that's important to you, the command line launcher approach perhaps makes it a little less likely that they will discover the mechanism - knowing that there are two executables isn't enough - they would also need to know the command line switch that your command line launcher adds - if they just used myappgui.exe directly from the command line, without using the command line launcher myapp.exe, then it is still going to respond as if it were launched from the GUI!
On the other hand, using a switch to indicate a GUI launch necessitates putting that required switch on "public view".