The straight forward answer is that you cannot tell how the JVM was launched.
But for the example use-case in your question, you don't really need to know how the JVM was launched. What you really need to know is whether the user will see a message written to the console. And the way to do that would be something like this:
if (!file.exists()) {
Console console = System.console();
if (console != null) {
console.format("File doesn't exists%n");
} else if ( /* we have a GUI Huston */ ) {
JOptionPane.showMessage(null, "File doesn't exists");
} else {
// Put it in the log
}
}
The javadoc for Console, while not water tight, strongly hints that a Console object (if it exists) writes to a console and cannot be redirected.
I'll leave it to someone else to fill in a suitable "we have a GUI" predicate ...
EDIT ... and @Stephen Denne provides the answer !GraphicsEnvironment.isHeadless()
.