When you say "calling it from Java", are you shelling out to Matlab for batch computations, or do you want to embed a long-lived Matlab session in your process and call M code repeatedly from Java code? What OSes do you want to run on?
Matlab has some deployment tools that let you embed a Matlab interpreter and a collection of Matlab source code inside a host language, such as C/C++ or Java. This is what the "Matlab compiler" is - not a real compiler, but a tool that packages a Matlab runtime along with .m source code in a package that looks like a DLL or application. A Matlab "standalone application" is Matlab code that has been packaged this way along with a thin C wrapper that calls an application entry point in your M code.
The Matlab Java Builder is a similar thing that bundles this deployed Matlab engine inside a Java class. If you want to get a license for it, that could make it easy and cosmetically clean to embed Matlab inside your Java application. This is probably what you want.
These deployed Matlab apps do not have a command window because they're intended to blend in with your application. They live in the same process. And, importantly, they do not require license fees for running the deployed app. Shelling out to regular Matlab requires all users running it to have licenses for Matlab and each toolbox that is used.
If shelling out, the "matlab -nosplash -nodesktop" command line will suppress the GUI on Unix. But on Windows you'll still get a minimal Matlab command window. The "-automation" switch on Windows will at least make it minimized. I don't know a way to suppress it entirely on startup.
However, once Matlab is running, you can take advantage of the fact that the Matlab GUI is itself implemented in Java, and have it hide itself. Get your Matlab session to run this hidematlab() using the "-r" command line switch or a startup.m. Note that this is a hack using undocumented Matlab internals and is surely unsupported by MathWorks.
function hidematlab()
%HIDEMATLAB Hide the main Matlab desktop window (HACK)
dtWin = desktopwindow();
if ~isempty(dtWin)
dtWin.setVisible(0);
end
function out = desktopwindow()
%DESKTOPWINDOW Find the main Matlab desktop window (HACK)
wins = java.awt.Window.getOwnerlessWindows();
out = [];
for i = 1:numel(wins)
if isa(wins(i), 'com.mathworks.mde.desk.MLMainFrame')
out = wins(i);
return;
end
end
Beware of gotchas when shelling out on Windows, where Matlab is inherently a GUI app. If your M code throws errors that bubble up to the top level or segfaults, you may find your Matlab session hung, waiting for nonexistent user input, instead of returning you an error.